Topic: MapMate is Full

Hi all,

We have been using MapMate to exchange with our recording groups, and using Recorder to hold our botanical records and habitat data.
Our copy of MapMate has now reached 1,071,646,720 bytes ...I have not had any kind of error message from it when importing new records, however I am told that MapMate cannot hold more than 1GB, as it is based on Access 97.

Is there a point when MapMate actually tells you it's full? Has it been shedding records in order to let me put new ones in?!? (Please say no.)

Anyway, we are looking at how we can resolve this in the short and long term, have other MapMate using LRC's met with this issue? What have other LRC's done about it?
MapMate is very good for receiving records from some of our smaller groups, and we will want to retain the ability to exchange small numbers of records with them.

We are thinking of;
1. having more than one copy of MapMate and using a separate copy to exchange with the more prolific recording groups
2. moving some datasets from larger recording groups into Recorder and asking for updated lists from their databases each year -importing those directly into Recorder- or maybe importing their entire database again each year, so any corrections are also picked up.
3. ?

Thank you for any comments or advice,

Lizzy Peat
HBIC

Re: MapMate is Full

Have you tried compacting your database? (In MapMate click on 'Dataset' on the toolbar and choose Tools > Compact.)

I don't believe there is a satisfactory way to deal with the 1GB limit - there was a thread on the MapMate users group on yahoo recently about this with various 'solutions'.

I have a feeling that Access (and therefore MapMate?) just 'breaks' (can't open, hangs etc.) when it goes over the 1GB limit but I've never experienced it myself... if it's still working your probably okay, but you might want to check your backups...

Charlie Barnes
Information Officer
Greater Lincolnshire Nature Partnership

Re: MapMate is Full

Thanks Charlie,
Yes I compact and repair everytime I do any sync or import work, so no space to be gained from doing that.
Best wishes,

Lizzy Peat
HBIC

Re: MapMate is Full

Several people (including BSBI) on the Mapmate email group have been having the same problem. If you're a member, you can see the discussion here:

http://groups.yahoo.com/neo/groups/MM-U … opics/6452

This is the message from Martin Rand BSBI recorder for South Hants:

Nick, we had a discussion nationally in BSBI in 2004 about this looming issue, and we calculated that, depending on the average amount of Comment text included in records, MapMate's DB would break  at between 5 and 6 million records. So you've proved it; and I learn that Hampshire Biodiversity Information Centre are about to prove it too, with roughly 4,200 records and 0.8Gb consumed. (HBIC's main repository is Recorder but they have to maintain a MapMate database to exchange data with their many MapMate-using partners. )

If Mark is actually doing anything to address this then (a) it's taking him a heck of a long time to do it; (b) he's keeping it very close to his chest. Moving to a later version of Access won't help much, because although the limit has now gone to 2Gb, this will be mostly swallowed up by later versions using a text encoding that requires twice as much storage for each character. Migrating to a different database platform will be a nightmare, because the differences in SQL dialects will break most of the queries that MapMate itself, and its user base, have written - and some of the naming conventions that Access allows for tables and columns will not be acceptable in many other DBs, so that compounds the problem. Conversion tools are around, but you now have the job of getting them deployed to a large user base of varying technical abilities, and correctly applied against an unknown set of user SQL code some of which could be problematic.. In this situation I wouldn't want to be in Mark's shoes, and I think my strategy would be "do nothing". I do actually think that MM should be migrated to a later version of Access, because I'm nervous about the reliance on unsupported 17-year-old software, and it irks me that I have to jump through hoops to handle MM data within MS-Access tools (I realise that latter isn't an issue for most people). I'm not sure what the problem would be with that migration - technical issues are few if any, and there's no licensing issue. Probably the effort - I would guess it's highly improbable it could be funded through sales revenue.

But the size thing is going to be a non-problem for all but the big organisational users (and Nick). And if they are using MapMate for a large national or regional archive then they probably need to be rethinking their IT strategy. When MM is used as a "staging post", it's not really too difficult for such users to find a way of partitioning the data set to overcome the limitation.

- Martin Rand
  BSBI Recorder, South Hants (VC11)

There are some ideas for workarounds in that thread, but they look a little hairy. We use an approach similar to your #2 suggestion. We sync in data, export it out to Recorder, then archive it. I've documented the process I hacked together here: https://github.com/charlesroper/mapmate-to-recorder (see the Word doc). It's written as a reminder to myself more than anything, so includes stuff that is specific to SxBRC, but the principles are transferable. I refer to a "cleaner script" in there which isn't uploaded to GitHub as it's very specific to Sussex data and includes recorder names. I can pass that on separately if you like - just ask. It's coded in Ruby so you'd need Ruby installed if you want to make use of it. That said, the need for the cleaner script is greatly diminished now that in the latest version of Recorder we can import recorders as an unmatched string.

Re: MapMate is Full

Thank you very much Charles,

This message on the MapMate forum was posted after I had an interesting discusion with Martin Rand directly, of 'the end is nigh' variety. I also did not like the solutions presented by other MapMate users. Martin tells me that BSBI nationally has had to set up a series of partitioned databases to handle the volume of records they now get from the county recorders using MapMate, and they did this as MapMate did not seem to be looking into upgrading the system.

Ruby coding is beyond me at the moment, but Wolfgang will be looking into this further for HBIC and may be in touch with further questions for you!
Thanks again,
kind Regards,

Lizzy Peat
HBIC

Re: MapMate is Full

Hi all, thought I should sum up what we did in the end-

We decided to move all Lepidoptera to a second copy of MapMate, and then delete the records from the original- we will now exchange lepidoptera only using this new MapMate.
We have also decided that we will start putting bird records onto Recorder from now on (by year) as our bird group only use mapmate to send us records- so this is much better for them, and will free up more space.

It has made some querys much trickier- if someone wants all records for a parish (rather than just notable species) we now have to quiz three databases.
However it does give us a breathing space.

Kind Regards,

Lizzy Peat
HBIC