Topic: Living Record Online Recording

Hi All,

The following has come through from Adrian Bicker of Living Record and may be of interest to those considering an online recording implementation.

LRC Manager,

If you do not have online recording or want to augment your existing system, we would like to help you.
We are encouraging LRCs to use Living Record, including

Tom Hunt - ALERC National Coordinator

Re: Living Record Online Recording

It's really great to be witnessing at last a great bloom in online recording systems. When I joined SxBRC 10 years ago, it was my hope that we'd see online data entry and online shared databases as the norm. It's fantastic to see it actually happening now. This is Biological Recording 2.0.

Something that I think should be discussed now is the flow of data between these systems. While the bloom is great for diversity and inspiration and suiting various needs, I am noticing a level of anxiety among recorders as to which data entry system they should choose to the extent that they're avoiding all of them. This is The Paradox of Choice (Edit: here's the TED talk - worth watching). It's also going to become more of a burden for upstream handlers such as LRCs, local groups, VCRs, NSSs, who will have to ensure they check each of the various systems. Things are getting messy fast, and the productivity we gain from having data go straight into online systems could easily be lost again by having to manage so many inputs and variable.

So what to do?

I think the major players in this area - Indicia/iRecord/BRC, Living Record, BTO/Wildlife Trusts/WildWalks, Rodis, some key LRCS, NBN, etc - need to get together and work out a sharing protocol so that data entered into one system can be synchronised with the others. This will allows for pressured VCRs, LRCs and other upstream handlers to use the system they're comfortable with, while also ameliorating the anxiety of choosing one system over the other. For instance, we use iRecord and Indicia here in Sussex. We'd like our local groups and recorders to use whatever system they're most comfortable with or is most suited to their needs. If that is Living Record, or BirdTrack, or a customised Indicia form then great. What is critical is that records entered into, say, Living Record show up in iRecord's verification system and vice-versa.

I know iRecord can work with records from any third-party site built with Indicia, but what are the prospects of similar 2-way sync with Living Record and the others? I hear through the grapevine that work is underway to enable this sort of sync with BirdTrack and another Wildlife Trusts funded as-yet-unannounced project developed by BTO. These new Living Record LRC Services would fit in great with this kind of automatic data-sharing.

Does anyone else have any thoughts on this matter? Are there any downsides to syncing data in this way? Would be interesting to hear Adrian's thought (who I know has just joined the forum) and perhaps thoughts from John van Breda or David Roy from iRecord/Indicia/BRC, and perhaps from Luck at Cheshire? Anyone else have a bespoke ORS?

Re: Living Record Online Recording

Hi Charles
I fully agree, the ability to easily exchange data between systems, in particular to allow verification, would be a really useful development. There are a few issues we need to take care about when exchanging records like this (terms and conditions being met, data protection) but mostly they are things that we can work around. As I understand it the development of a generic sync API plus a trial implementation between Indicia & Birdtrack is still planned but funding needs to be sorted out before it can proceed.

BTW I've got a Recorder 6 addin for pulling Indicia/iRecord records down on the way...

Best wishes
John

John van Breda
Biodiverse IT

Re: Living Record Online Recording

Great news re R6 addin John. Rodis and living record ones would be great too. (We had an email conversation a while back I think -Rodis one should be able to deal with individuals and locations as well).

For CBDC and online recording, data flow and efficiency of working with Recorder 6 is of much higher urgency and importance than verification, because that is an issue costing us time now, whereas verification and syncing will require us to change systems in the future (not that I am anti that in principle, it will just take time/resources).

Syncing - is birdtrack the best place to start? How typical a system is it compared to others? I am not sure, as this really taking me to the limit of my hazy database knowledge, but would it be fair to say that Birdtrack, iRecord and Cofnod's ORS are the only truly online databases where syncing may make sense, whereas all others are temporary collection databases? I can't seen why I would need to pull iRecord data into Rodis or vice versa?

On a related matter, I'll shortly be asking for volunteers to join an Alerc working group on NSS/LRC data exchange and flow. Online recording will of course be one thing for the group to look at, although by no means the only thing.

Teresa Frost
WeBS National Organiser, BTO. NFBR Council Member.
ex-ALERC/CBDC/KMBRC

Re: Living Record Online Recording

TeresaF wrote:

I can't seen why I would need to pull iRecord data into Rodis or vice versa?

Can your VCRs and other verifiers do verification work in Rodis? Do they, or could they, use other systems such as iRecord or Living Record to do verification work? The principle use for data sync is so that verifiers need not be forced to use one system over another. But it also greatly simplifies matters for pulling data into Recorder - it's much easier and less error prone developing a workflow that pulls from one upstream source than many.

For example, your workflow could be to pull data into Rodis from iRecord, Living Record, CORS, BirdTrack, etc. Some will be verified, some will not. Those records not verified in the other systems could be verified in Rodis using your locally preferred workflow. Once you are ready, you could then pull data into Recorder from Rodis for further verification work and use in your day-to-day operations.

But what of data that is synced into iRecord from other ORS? Doesn't verified data on iRecord automatically pass on to the NBNG? Yes and no. Data entered into iRecord or through one of the iRecord-provided custom forms directly is administered by BRC and under that agreement verified records are passed to the NBNG. HOWEVER, data synced into the iRecord community warehouse from external sources (i.e., custom Indicia forms at present) are administered by the form owner and do not automatically get passed to the NBNG. So records can be viewed, queried and verified by the recording community in iRecord, but the appropriate custodianship is maintained along with terms of use and established data flow, etc.

Re: Living Record Online Recording

Hi Charles, I am currently working with Adrian Bicker and Living Record in Hampshire.

In our view, Living Record is an ideal way for our species groups to receive records quickly and easily from may different people- we all treat living record like an email inbox, but without the faff. Once records are swept up by our county recorders, they add them to their existing databases where the records are edited/maintained as usual. We continue to exchange with the groups using MapMate, which works very well.

The Living Record mapping is great for getting good grid references, and the volume of records received by our groups is increasing everyday as people get into it. However, Living record it is not to be used as a separate database system, it is a 'shunting records along to the groups and lrc's' system. Once downloaded, records are 'batched up' in Living record and won't get downloaded again, so duplication is not a problem. Also it is worth bearing in mind that records on living record have not yet been verified- this all happens in the species group systems as usual.

We are currently working on creating a download format which can be imported directly into MapMate (once saved as a tab-text file.) Adrian and I need to work out which of the masses of recording methods/stage/status currently available in MapMate are important, and should be available. At the moment we are looking at the Mammals and Amphibians and reptiles specifically, as our ARG and mammal group are using living record now, and this would make their lives easier. This has already been done for our BSBI recorders.

If there isn't anyone covering a particular species group, I am able to sweep them up directly using the LRC services. It's early days, but looking good. Happy to talk to anyone about it if you want to give me a ring/email.

Lizzy Peat
HBIC

Re: Living Record Online Recording

They could, in that it is a rodis feature, but they don't, because it doesn't make sense for us or them at the moment - most data goes straight into Recorder. So we'll always need to verify outside an offline system, hence it is just as easy to do that within our offline systems as the online system - that way we can use the records in the meantime without worrying about duplication.

Last time I checked, no Cumbrian recorders are verifying iRecord data or Rodis data. Most of our verifiers work from spreadsheets and look on an annual or monthly basis, some come into the office to look through stuff. 

Rodis is different from iRecord, in that as an LRC I don't do a download of everything like you do in iRecord. Instead the recorder has their own files (like R6 surveys) and when they are happy with them they submit them to the LRC. Once that is done, it is only a copy on Rodis that the recorder cannot change, but can still download/view/map/see summary statistics of. So very like Living Record from what Lizzy has said. Think of it more like an online version of SXBRC's Species Recorder with a few extra bells and whistles - which by the way we used at our bioblitz very successfully at the weekend, as no signal meant I couldn't use Rodis. smile

Teresa Frost
WeBS National Organiser, BTO. NFBR Council Member.
ex-ALERC/CBDC/KMBRC

Re: Living Record Online Recording

Hi Lizzy, many thanks - that's useful info. So you can't do verification work within Living Record and recorders can't use it for managing their personal records - it's purely a data entry system?

How do you handle records coming from iRecord, BirdTrack, new Wildlife Trusts systems, etc.? What happens if a recorder wants to edit/access data that has been downloaded from Living Record?

It sounds like pulling data into Living Record from iRecord isn't feasible (because data on iRecord is editable), but doing it the other way would work, as is the case with, say, iSpot data. Groups (local and national) can then use the iRecord facilities to do their verification work. It may be that some groups within Sussex would use Living Record, but I'd like to be sure data could be easily pulled into the BRC community warehouse for verification work in iRecord and subsequently downloaded by us.

Re: Living Record Online Recording

Teresa, same here, the vast majority of verification work in Sussex is done offline using a group's own processes or our own processes with involvement from a VCR. This kind of workflow is fraught with practical problems and complexities though, so is hardly ideal. My hope is that, gradually, more verification work can be done upstream within the online systems. As recorders see the benefits of using such systems, more will be encouraged to participate that way. Safe to say it's a long term aim, but we'll get there in the end I think. :-)

Re: Living Record Online Recording

We have been verifying records effectively in Living Record for the last two years and the verifiers seem happy with this approach. Verified records are batched and downloaded, either by the verifier for his VC(s) or by the LRC using 'LRC Services'. The LRC can download data for a VC or for their own LRC area, cutting acress VC boundaries. So the general approach is that only 'good' records come out of Living Record.

Verification in Living Record aims not to get in the way. Clearly the relationship between verifier and recorder and a good understanding of the recorder's abilities and limitations is the key to assessing the quality of the records. While some verifiers still think of verification as an annual task, others have realised that it is a great way to keep in touch with their recorders - to thank and encourage them. Some review records daily so that they know who is finding what where. If there are any questions about my moth records, I generally know by lunch time!

The main Living Record download format has been developed working with Alison Stewart at DERC for easy import into Recorder 6.
Alternative download formats are provided where necessary e.g. a Levana downoad for butterflies and a MapMate download for vascular plants. This was added last year for the Hants Plants team. As Lizzy says, we will be adding MapMate download formats for other subjects soon.

Alison reports that the downloads from Living Record save her a lot of time and effort due to their consistency and high quality. If people send casual records to DERC, DERC volunteers or staff generally enter these direct into Living Record. It is faster and easier than entering them into Recorder 6 and this way, they join the main flow of records through online verification and through to the downloads.

Charles, a recorder can amend or delete any of his records up to the point when the verified records are batched for download (usually some time later). Batching locks the records. This ensures that the download will give consistent results if it is repeated. Note that a verified record that is amended by the recorder goes back to being a 'new' record and has to be verified again.

Adrian

Adrian Bicker
Living Record

Re: Living Record Online Recording

Thanks Adrian for the explanation. I agree with your comments about verification - online systems should make it easy and even fun to engage in verification, even on a daily basis. That's partly why I'm keen for verifiers to be able to use the system they're most familiar and comfortable with. And the good news is it seems that a two-way sync with other systems would indeed be possible?

I'll ask, since you're here: would you be willing to collaborate with the makers of the other online recording systems to implement syncing? Download formats are not the same as sync. As far as I know, all ORS offer download in a variety of formats for easy import into local databases. It is sync, both from and to other ORS and, selectively, with Recorder and I suppose MapMate that would be the real time and effort saver.

As John said, BTO are working on a sync API for Indicia/BirdTrack and the upcoming Wildlife Trusts system, but we really need commitment from other developers so that we can ensure recorders, verifiers, LRCs and so on are able to use the ORS they are comfortable with and not be forced to use one over the other.

I asked in my initial response: "Does anyone else have any thoughts on this matter? Are there any downsides to syncing data in this way?" Maybe I'm missing a key point here, but why would an ORS not want to selectively sync with others?

Re: Living Record Online Recording

charlesroper wrote:

How do you handle records coming from iRecord, BirdTrack, new Wildlife Trusts systems, etc.? What happens if a recorder wants to edit/access data that has been downloaded from Living Record?

Hi Charles,

In answer to your question above, our species groups handle this work themselves- I think the Hampshire Ornithological Society (HOS) are actually using Cobra as their main database at the moment, they put records from all these different systems into MapMate, which they use to Sync with us. We then make this into a GIS Layer containing all notable protected and notable species, which is provided to our partners and district councils.

If anyone wants to query the data they come to us, we deal with all enquiries on behalf of HOS, and provide them with a proportion of the income generated. Any corrections which are needed are done in MapMate by HOS and then an update sent back to us. Once records have been taken from living record, our view is that they have achieved their purpose and can be safely ignored until deleted. So there may be incorrect data on Living Record, but as it is a conduit it is not a problem.

Kind Regards,

Lizzy Peat
HBIC

Re: Living Record Online Recording

Thanks Lizzy. So you don't handle downloads from iRecord or the others directly, only Living Record? I can see you've got a fair amount of data for Hampshire in iRecord across a broad range of groups and coming from a number of surveys (Plantlife's Wild Flowers Count, the Ladybird Survey, Garden Bioblitz, BWARS, Moths, Plant Tracker app, etc.) There are presently over 5000 records for N. Hampshire and over 5000 for South too, so it's a substantial and useful dataset.

Re: Living Record Online Recording

charlesroper wrote:

But what of data that is synced into iRecord from other ORS? Doesn't verified data on iRecord automatically pass on to the NBNG? Yes and no. Data entered into iRecord or through one of the iRecord-provided custom forms directly is administered by BRC and under that agreement verified records are passed to the NBNG. HOWEVER, data synced into the iRecord community warehouse from external sources (i.e., custom Indicia forms at present) are administered by the form owner and do not automatically get passed to the NBNG. So records can be viewed, queried and verified by the recording community in iRecord, but the appropriate custodianship is maintained along with terms of use and established data flow, etc.

Thinking aloud/exploring further:
If we decided all our records, from all sources, would be verified online through the iRecord warehouse, would that mean all iRecord users could see all the sensitive and recorder info, full res grid refs, comments etc or could you make records only viewable to the verifiers?
Could we then stop syncing them after a certain amount of time? Should we?
If we could all view all records through iRecord, the NBNG becomes defunct?
"verified by the recording community" I like iSpot very much, and it has an innovative approach to "crowd-sourced" IDs, but it doesn't always get it right. Certainly I know of frustrated users when someone with a higher score has incorrectly changed the top determination of a sighting. iSpot is not for recording so that doesn't matter and the data custodian would presumably have final jurisdiction for other ORS data being pulled into iRecord.
A sync for iRecord data with R6 (not other data in R6) might be interesting, and help with the issue of what to do with iRecord* data now. i.e. the conundrum of - either we add it to our database and not verify it, perhaps flagging as "not locally verified", passing only to local authorities and consultants, but not recorders, or we commit resources to getting feedback from verifiers, changing the verification on iRecord ourselves, and then re-download, or we ignore it and thus deprive data users of valuable records. (We can't send it to our verifiers and just make changes on our local system, as that would be against iRecord T&Cs.) But if I changed the determination in my copy of R6 I could be over-ruled by someone else on iRecord, would that work??

*by which I probably mean entered into iRecord, not planttracker etc.

I only have questions don't I, never any answers or possible solutions. smile

Teresa Frost
WeBS National Organiser, BTO. NFBR Council Member.
ex-ALERC/CBDC/KMBRC

Re: Living Record Online Recording

I can see that a shared verification service could serve a useful purpose, especially as the flow of records and the number of online systems increase.

To be widely accepted a shared verification service should be:

1. Neutral and not allied with any of the online recording systems.
This enables all online systems to use the service without compromising their independence or going against the wishes of their users.

2. Separate from BRC and the NBN.
We are keen to enable LRCs and NSS to retain their independence and to keep control over their own data. LRCs should still be able to decide what data to put on the NBN Gateway, when, at what resolution etc. Fast-tracking records into the NBN Gateway or publishing them in iRecord seems to undermine this independence and could ultimately enable BRC/NBN to control the whole recording cycle, from entry of records to publication on the NBN Gateway.
Many recorders would not wish to submit online records if the way that they are held or verified undermines the work of their LRC (or NSS).

Adrian Bicker
Living Record

Re: Living Record Online Recording

Hi Teresa
The question I can easily answer from the above - iRecord has functionality to expose datasets for verification but to not expose the records for public users/reporting or downloads. Or we could expose the records for reporting but blur to a larger grid ref and hide the site name.

In iRecord things are slightly different to Living Record in that we envisage that the top copy of the record always stays in the possession of the recorder, on iRecord. That means that if you add a determination to a downloaded record in Recorder 6, it would be "polite" to sync that determination back to iRecord so that the recorder is notified. Or potentially to make the determination in iRecord and resync into Recorder 6.

Given that PlantTracker records (and other recording scheme data collected using Indicia websites) are being verified on iRecord I'm not quite sure why you would distinguish to download only the iRecord input data into Recorder 6? Unless you decide of course that a particular's scheme's data is not of interest.

I agree with Charles' earlier point - even if Living Record is not a data repository, there could be benefit to syncing iRecord records into it so that verifiers can use their chosen tool to do their verification rather than have to switch tools. The records would be effectively only a transient copy on Living Record that would be synced back into iRecord when verified (and could potentially be deleted at that point from LR). I also agree with the point about online verification having great potential to change the way that verifiers work - if an online verification tool can make the verification process simpler and quicker (which I believe they do) then we will gradually see a shift away from spreadsheets and complex data flows. Why would I want to download into a spreadsheet when the online tool could give me the same functionality, plus Record Cleaner rule integration, batch verification, trusted recorders, managed communications etc etc?

Cheers
John

John van Breda
Biodiverse IT

Re: Living Record Online Recording

Thanks John, all very helpful points. I only ask a lot of questions because I haven't quite got my head around the iRecord concept or indeed online databases in general. I meant it would be iRecord input data I would have more reservations about downloading at the current time. In my old school way of thinking, I'd consider the data coming from particular schemes, eg plant tracker where I believe a lot of work is done with photos etc, is no different from any other verified dataset administered by another organisation. General iRecord records are not a wholly verified dataset, so it is these I am more cautious of - but I realise there is an intention with iRecord to consider verification on a record by record basis so perhaps I need to adapt my thinking a bit.

A lot of aspects of iRecord is quite a big philosophical change to traditional recording approaches e.g. the top copy of the record staying with the original recorder, regardless of whether they are a traditional recorder or a hill walker who wants to report a waxcap in response to a newspaper article say. I only seek to understand so I can do my job of supporting recorders as best I can, widen participation in recording and the amount of available data, but at the same time considering the needs of my data users (particularly planners and developers) who need information to be as robust as possible, as the wrong information could cost thousands in consultancy or legal fees or just lead to the wrong decision to be made.

On the politeness of letting the recorder know - that is a very interesting point as well. At the moment I wouldn't necessarily tell public survey type recorders if their records had been marked as incorrect. e.g. Big butterfly count records. Unlike some LRCs, who get all their data via VCRs or societies, almost all of CBDC's comes to us from individuals so it would be us that would have to tell them. It might dishearten people; on the other hand it might encourage them to learn. It might open up nasty disputes between recorder and verifier that we would get in the middle of. It might make the data better as a result. Much to ponder. smile

I agree with Adrian any verification shared service ideally shouldn't be coming from "this is how you can work with my system" - is such a thing technically possible though? I am well beyond my knowledge here, but would it have to be a peer-to-peer type thing, like bitcoin or a music service, so that the data wasn't being held on someone else's server? If you extended the NBN data exchange format, and then I was top Cumbrian mite expert who liked living record's interface, could I pull in data from irecord, living record and rodis and verify it and send it back without it ever having been on living record's database?? It is no doubt very obvious I haven't got a clue what I'm on about so I will shut up now.

Teresa Frost
WeBS National Organiser, BTO. NFBR Council Member.
ex-ALERC/CBDC/KMBRC

Re: Living Record Online Recording

I am going to reply to this thread eventually.-Having a particularly hectic week here and this discussion deserves some proper attention, so it's taking longer to come back with a reply than I'd hoped. :-)

Re: Living Record Online Recording

Just to add, we have developed a system at BIS for the recorders to see all BIS records relevant to their county and taxon group on-line. So they can verify historic data. At the moment BIS has to exchange any records received over the year with the VC recorders using spreadshees which they can verify. Needless to say we have a lot of historical  unassessed records as we do not have many county recorders and cover a large area. I hope direct access to a web service will allow them to view, verify and download the records at their leisure and lessen the workload for BIS.
I did wonder with Living record that it appears to rely on the records being verified before they can be downloaded. As this can be quite demanding for individual county recorders I would want to download unverified records if need be, otherwise they could sit there for a long time. I can also see a problem for all these on-line sysems that if they changed the verication after a download you either download again with a new verification and therefore a duplicate record, or you do not get the changed verification on the record. If these on-line systems had a TOK then you would not get these problems as it would update the record on import, but you cannot import a TOK into R6. Or I suppose you just report directly from NBN Gateway where all these records will end up. Anyway just thinking on....

Janet Imlach (Manager)
Powys and BBNP Biodiversity Information Service

Re: Living Record Online Recording

Hi  - I just wanted to clarify a couple of points about data flow via iRecord/Indicia systems to the NBN Gateway.

The ability to

Re: Living Record Online Recording

Hi Paula

Does this mean that LRCs that standardise on Indicia/iRecord to collect local data will no longer have to send datasets to the NBN Gateway - because the verified Indicia/iRecord data will go onto the Gateway anyway, attributed to the appropriate national society?

Thanks
Adrian

Adrian Bicker
Living Record

Re: Living Record Online Recording

Hi Janet

We recognise that a LRC should have the option to download all records from Living Record without them being verified online. We don't want records to get stuck in Living Record.  This feature will be added in due course.

However, this approach does put the onus on the LRC to take responsibility for avoiding duplicates and omissions in subsequent downloads.  We have some ideas to help with this too.

The ability to create batches of verified records in Living Record is a tidy approach and makes it simpler to manage the download of data.  Batching locks records, preventing significant amendment or deletion so that you get consistent results whenever you download a batch of records.  This seems to work well for verifiers, like your own Bob D.

Some county recorders verify, batch and download records frequently.  Hants Plants is a good example as they top up their county MapMate database with records from Living Record most weeks.  So the LRC may be faced with multiple batches of verified data.

Do not worry!  LRC Services lets you combine multiple batches (across multiple VCs) into a

Adrian Bicker
Living Record

Re: Living Record Online Recording

Hi Adrian (and all!)
LRCs are likely to continue to supply datasets to the NBN Gateway when:
a) they have received records 'offline' and the local data flow agreement is that the LRC (rather than the NSS) disseminates these records for wider use
b) they have set up their own Indicia website / iRecord survey and wish to administer the resulting data on the Gateway themselves
We're about to start testing the new Gateway admin controls with a small group of users including several LRCs, with the aim of ensuring that dataset administration is as easy as possible and not a burden - for whoever is doing it!
thanks
Paula

Re: Living Record Online Recording

Hi Paula

Could you clarify this please.

If a LRC sets up an Indicia web site, is it just the data that is collected through that web site (or iRecord survey) that the LRC can submit to the Gateway?

What happens to all the other records in the Indicia/iRecord database that fall within the LRC area?  Does the LRC have any control over how and when that data is submitted to the Gateway?

Regards
Adrian

Adrian Bicker
Living Record

Re: Living Record Online Recording

Adrian, I can answer these for you.

1. If an LRC sets up an Indicia form/survey/website, whether or not it is linked to the BRC Community Warehouse and/or iRecord, then they are the custodian of the data collected and therefore can submit data to the Gateway or to a NSS or a local group as appropriate.

2. Data entered by individuals directly into the iRecord standard data entry forms come under custodianship of BRC and not an LRC, therefore the LRC does not have control over how this data flows to the NBNG, although the LRC can download and use the data on-demand. If the LRC requires absolute control over how data is provided to the NBNG then the generic iRecord data entry forms should not be used. If it is the case that the LRC absolutely requires control over data flow then one of the alternatives should be chosen (e.g., a custom Indicia form, custom iRecord form, Living Record, Rodis, etc., etc.)

Also don't forget that because Indicia is completely free and open source, one can easily set up an entirely independent server and or client website.

"Does this mean that LRCs that standardise on Indicia/iRecord to collect local data"

Talking in terms of standardising on Indicia/iRecord is the wrong way of looking at it. My plea to the makers of ORS is, through agreement, to develop protocols for seamlessly exchanging data between those systems. So in theory an LRC would standardise on the protocol, not the individual product. This is a healthy way to proceed. I think it is important to have a diverse range of products out there, including Living Record, Indicia, iRecord, Rodis, BirdTrack, etc., but these system should be striving for seamless interoperability. This would mean that an LRC wouldn't be forced to use one system over another. Individual recorders wouldn't be forced to use one over the other. Groups wouldn't be forced to use one over the other. People could use whatever system most fitted their need, creating a healthy, diverse ecosystem and mitigating proprietary lock-in.

For us here in Sussex the right fit is a combination of iRecord for individuals looking for an all-round ORS for managing their own data. We supplement that with custom Indicia recording forms that either link to the BRC Community Warehouse so that they can benefit from the verification system there, or not as data flow dictates (this is theoretical at present as we haven't developed any custom forms yet, but these are coming). As Paula wrote, the way Indicia, iRecord and the BRC Community Warehouse interact is very flexible and controllable. We also have our own completely independent Indicia warehouse for use with our own LRC Online reporting system. We could link custom input forms to that, too, if necessary, but it's not as BRC do an excellent job of running the BRC Community Warehouse. BUT we also want recorders to use what they are comfortable with; if that is Living Record, or BirdTrack, or whatever, then so be it. We don't want to force anyone to use any particular system. It should be our job, and the job of ORS developers, to ease that burden of choice.

Finally, Adrian, I'm not sure what you mean by a shared verification service that is neutral. I am talking about a protocol for sharing data between servers, not one monolithic shared server. This would mean in signing-up for one service there would need to be agreement to share your data with other trusted systems (much like social networks ask for permission to share data), and of course an option to opt-out of such sharing should the user wish.

Re: Living Record Online Recording

Thanks Charles, this is very helpful.

I meet such nice people on this forum ;0)

The approach in Living Record is rather different

Adrian Bicker
Living Record

Re: Living Record Online Recording

ADRIANB wrote:

The approach in Living Record is rather different

Teresa Frost
WeBS National Organiser, BTO. NFBR Council Member.
ex-ALERC/CBDC/KMBRC

Re: Living Record Online Recording

> "Thanks Charles, this is very helpful.

I meet such nice people on this forum ;0)"

We aim to please. :-)

> "It is hard to square this intimate co-operation between friends with the automated transmission of data to other national databases.  It could feel too Big Brotherish for many."

This is true for many, but the inverse is true for many too. You only have glance around the web to see how many, many people are hugely eager to share their data and information, whether that be photos, stories, music, videos, travel locations, favourite x, what they had for lunch, etc., etc. So naturally the open collaboration and free flow of species data is a powerful motivator for some. In doing training workshops for iRecord - all local people and passionate about their patch - very few people were concerned by the public nature of the data. There were concerns about sensitive records being entered, but these concerns are ameliorated by the NBN Record Cleaner auto-checks and the ability to 'blur' records to 2, 10 or even 100KM. The fact that only verified records pass to the NBN Gateway has also been a big reassurance, with the main concern there being that 'rubbish' data could easily make its way into the national database. The biggest concern - by far, it would seem - is the anxiety about which system to use for which purpose, which is why I bang on about interoperability. There is a natural reluctance to fully adopt any new system where the detail of how it fits into the bigger picture is uncertain.

The Big Brother issue - i.e., lack of trust for whatever reason - is an issue that does need to be overcome. Even at LRC and local group level we are the subject of similar suspicions from some individuals and organisations. Every LRC will have some experience of this. That's why options are important, and as much control is provided to the user.

> "There are also serious privacy issues and we need to ensure that people are not discouraged from submitting records because they worry about who can see their name, guess where they live or discover where they were on certain days."

Yes, it is very important that people understand the manner and detail of what and how they are sharing with the world. That's doesn't mean open sharing should automatically be discouraged or seen as a negative - on the contrary, it's a good thing that makes people generally feel good on both ends of the transaction. But people should definitely be provided with enough information to make an informed choice. One little technique on some sites (can't remember where now) is the ability for a user to set a radius around their home and if they create a data point (take a photo I think it was) within that radius, it warns them. Neat and effective. Alternatively, you can create sophisticated rules about who can see what, like on Facebook, Google+ and Flickr. Any privacy rules established by the user on a given system would need to transfer to other systems and be honored there too, should a sync mechanism be in place.

> "I did not realise that you have set up your own private Indicia database in Sussex. That option had not occurred to me."

Yes, it's currently used for reporting data only available to our partners. In becoming a partner of SxBRC, you gain access to our online reporting tools (LRC Online). Saves the partner from using or even requiring a GIS, which had previously been the case, therefore it can now be used more widely within a partner organisation. It runs on a [url=http://www.memset.com/dedicated-servers/vps/]

Re: Living Record Online Recording

Teresa, that's a great, clear analysis and I agree with it all. It's worth emphasising again the principle of iRecord data being a top-copy - you're not really "sending" records anywhere. The data is yours and you can edit or delete as you like. That's a useful distinction I think.

Another factor, especially with online services, is security and longevity - if I as a recorder adopt one system, can I be sure it'll be around in future? Can I be confident the server is secure and well run? What if the service shuts down? Is the business model sustainable? This isn't such a problem if the system works as a temporary holding-pen for records to be passed on, but is a concern if the data is to remain and be used and relied upon in some fashion and should definitely be topics for discussion and debate.

I'd also be reluctant to use the term "LRC-led ORS". This suggests iRecord has had no steer from LRCs when in fact it has (very much so), and it also evokes a "them and us" feel which is I think is very important to avoid. The term "LRC-controlled ORS" might be more precise here. Neither iRecord or Living Record could be strictly defined as LRC-controlled, but both are developed in collaboration with LRCs and are in many ways led by our requirements but with somewhat differing philosophies and emphasis. The distinction lies in how data custodianship and flow are defined; one is more liberal, the other more conservative. Rodis is definitely LRC-controlled, as is Cofnod's ORS. Locally installed databases can be LRC controlled (well, the data at any rate). Custom Indicia sites and private warehouses can also be LRC-controlled and have the side-benefit of being able to link in a controlled way to iRecord; and of course Indicia is Open Source with all of the benefits that confers.

Re: Living Record Online Recording

Another thought on the privacy issue: is it not better to establish permission and understanding and enable personal privacy control at the point of entry? No system can ensure personal privacy if the data is passed on unless the personal data is removed, therefore I'd argue that it is safer to establish permission and a clear understanding from the outset.

Re: Living Record Online Recording

I have been meaning to contribute to this discussion for some time - Cofnod's ORS is probably the earliest 'LRC - controlled ORS' (to use Charles's term) with its first version launched in 2006, and therefore many of the issues being discussed here have been thought about by us for some time. In fact our experience was fed into an early 'ideas gathering exercise' before iRecord was developed. It is very interesting to see how other systems are now developing and how the same key issues are being raised, e.g. how data is shared with other databases (online and offline), as we have found that this is rarely as straightforward as you might expect and needs some serious thought put into it - with this in mind I am keen to be part of of the Alerc working group on NSS/LRC data exchange and flow. Just to make a couple of points:

Cofnod's ORS is an online data management system (just to correct an earlier point) - the top copy is held on the ORS and in fact Cofnod are in the process of moving our entire data holdings onto the ORS. It is used by many individual recorders, a number of county recorders and others as their sole data management system - both individually and in groups, e.g. for certain monitoring projects where volunteers submit records using customised data entry forms and the information is shared within the group. The concept of it being top copy has been key throughout its development and something we are continually stressing to those to whom we provide access, including County Recorders, who often may wish to hold a copy of relevant data in their own databases, sometimes producing knock-on data management effects for them in how they exchange data with their National Scheme and Society.

County Recorders have always been key to the Cofnod ORS, as online verification has been part of the system from the start, with local experts (mostly County Recorders representing National Schemes and Societies) able to access and verify records almost instantly (records are assigned automatically overnight). At the end of last year we also built in checks based on the national verification rule sets used in the NBN Record Cleaner which has allowed us to prioritise which records local experts must check before they can be used. We have a 6 point system of verification levels which mean that records can be used even if they have not been individually checked (they are flagged approproately to data users, so long as they pass the automated checks and have a relatively low level of ID difficulty. We have built up quite a bit of experience in developing and managing online verification and would be keen to be part of national developments in this.

Aisling May
Data Manager - Cofnod   www.cofnod.org.uk

Re: Living Record Online Recording

Thanks for the input Aisling - it's good to know a little more about Cofnod's system. Perhaps now is the time to set up an ORS group (just an email discussion group would do) that anyone could join to discuss these thorny issues, share knowledge, ideas, code, etc. and perhaps get together in person from time to time. Who else here feels they might benefit and get involved in something like that? I can think of several topics worthy of discussion. The shared experiences of Cofnod, of BRC, of DERC and Adrian, of RECORD and its Rodis users, iSpot, BTO, WTs, NT, the multitude of Indicia implementors, plus Dan Jones and his experiences with online systems (not sure if you're working on an ORS, Dan) and John van Breda and other Indicia developers would be invaluable. Paula at NBN also has a keen interest and a wealth of knowledge and experience. Feels like everyone would benefit from a group collaboration like this.

Here are some hot-topics based on the discussion here:

* Data flow

* Verification

* Privacy

* Inter-ORS data sync

* Funding and business models

* Open Source vs. closed (pros and cons of both)

* Technology choices (PHP, .NET, Ruby, Python, PostgreSQL, PostGIS, MySQL, SQLServer, GeoServer, OpenLayers, Leaflet, Google Maps, OpenGeo, ArcGIS Server, HTML5, Flash, etc., etc.)

* Implementation showcases

* Collaboration opportunities

I can set this up, but who would be interested in participation? Even though we already have the technical forum here, technologists within LRCs rarely seem to talk to one another. Is that because there is so little common ground there isn't much to discuss? Or is it simply that people are heads-down busy with real work?

Re: Living Record Online Recording

I'd be happy to participate Charles.

To be clear about a couple of things. The very first brief for Indicia was for a data capture system where the records would flow through to other systems after capture, transferring the "top copy". Indicia can of course still be used this way. However we very quickly realised that Indicia had quite a lot of potential to act as the long term data store for some datasets where the complexity of something like Recorder 6 is not required. Also that many Recorders like to keep ownership of their records in one place. By keeping the top copy of records in iRecord it eases some of the problems surrounding duplications and data flows, though of course not all of them sad

And to confirm Charles summary of the sync options to Indicia/iRecord. There is of course a spreadsheet upload tool, but Charles mentioned Sync2Indicia, a generic tool which allows any ODBC dataset to by pushed up into an Indicia warehouse. There is also a Recorder 6 addin which will give a more or less single click pull down of records back into Recorder 6. This is in development at the moment.

It is hard to square this intimate co-operation between friends with the automated transmission of data to other national databases.

This is not what happens, apart from the general purpose forms on iRecord where all recorders have signed up to and agreed to this. Where we are collecting records for a local patch, small recording scheme or survey, we can give the organisers of that patch, scheme or survey the control over how the dataset is passed onwards, if at all. Or the organiser could agree that their records are included in the automated transmission to the NBN gateway (and at what level of public precision). We must also remember that for questions to be answered on things like climate change science, invasives, tree diseases or other wide scale issues, we need cohesive datasets on a national level (NBN) or even a worldwide level (GBIF).

John van Breda
Biodiverse IT

Re: Living Record Online Recording

Thanks all - Aisling I was wondering as I typed whether your ORS might be an online data management system, but I couldn't remember, so thanks for the clarification.

Teresa Frost
WeBS National Organiser, BTO. NFBR Council Member.
ex-ALERC/CBDC/KMBRC

Posts: 34

Pages 1

You must login or register to post a reply

ALERC Forum → General Discussion → Living Record Online Recording