After reducing the number of nodes of the islands I’m trying to upload, I have a better feeling that it goes well this time. However, when uploading, JOSM takes literally hours to upload my data, or, to be more precise, the “Postprocessing uploaded data …” step takes hours. For 1.000 objects, it takes about 20 minutes easily. For 5000+ objects, it takes two or more hours. I’m sure this wasn’t the case when I did some other uploads before my island project. Did anybody of you experience the same during the last days? Or is there something I can do to optimize my upload data?
Comment from Luciano on 17 June 2018 at 00:37
I remember seeing problems like this before, sporadically. Currently I am having no such problem, however. I think it must have to do with issues with how JOSM is configured on your machine, or (perhaps?) with problems with your ISP.
I just now uploaded 902 objects in about 20 seconds. Admittedly, South Korean internet speeds are sometimes spectacular - but that went all the way to Europe, of course.
Comment from zhenkang on 17 June 2018 at 04:52
When I moved Singkangia once and creating a new huge bunch of nodes, it can take 2 hours or so, I cant remember, because I just go out of the house while JOSM is still uploading, or went to sleep with my computer on.
It just depends how many nodes you are uploading. Just be patient.
Comment from EMKLI on 17 June 2018 at 07:51
Hmmm… I doubt that it has something to do with the speed of my internet provider (solid 4 MB/s LTE connection). And the upload itself is blazing fast. Takes only a few seconds for 5.000 objects. What takes hours is just the post processing. However, tonight I thought that it might have to do with the fact that I’m not only uploading these islands but on the same time adding them to two relations. Will make some tests about this.
Comment from EMKLI on 17 June 2018 at 07:55
I just recognized that obviously I didn’t add the islands to any relation. Possibly during one of the many hangups when preparing the data, that change went lost. So I can exclude this as a source of error too. I will try to download the beta version of JOSM and see if this is a version-related problem.
Comment from Luciano on 17 June 2018 at 10:37
As I mentioned, I HAVE seen this problem before. I can’t replicate it now. “Postprocessing” reads to me like server-side, but if it were server-side, it ought to be possible for other users to see it.
You might try deliberately “interrupting” your next upload when it drops into “postprocessing” and see what happens. You can do this by crashing JOSM (e.g. on Windows, CTRL-ALT-DEL to Task Manager and “kill” the process).
If doing that it breaks the changeset (i.e. lost objects), we know it’s server-side. On the other hand, if your changeset is fine in such a situation, we know JOSM is misbehaving somehow.
Another thought: how “old” is your JOSM (.OSM) file? I try to associate each of my uploads with a “fresh” .OSM file. I do this by having “workspace” .OSM files and then “upload” .OSM files. The upload file is created by going to the area to be edited, opening a new layer, downloading the current objects in that area, and then cutting and pasting new objects from my workspace file(s) into the “upload” file. Once the upload is complete, I close that upload file. So in most cases, each upload is associated with a “fresh” file. I don’t know if this is necessary, but i have found I have far fewer problems with JOSM by doing this.
Comment from ruadh on 17 June 2018 at 16:18
I had this problem, slow “post-processing” issue, last year when I moved territory. I was shifting large numbers of nodes (say 5000 plus) each time. Uploading the nodes was quick but for every node it took a few seconds of postprocessing. Played out over thousands of nodes made the whole process last for hours in some instances.
Never had the same issue with smaller numbers (around 1000). Something about big uploads can drastically increase the time it takes. I’ve a great internet connection, was running the latest version of Josm etc etc
Comment from EMKLI on 17 June 2018 at 17:35
Thanks for your support, Luciano. During the day, I’ve tried some things:
1. tried out the latest “untested” JSOM version - same effect
2. followed your approach with copying the objects I want to upload to a new OSM file (including saving) - same effect
3. Trying to upload parts of the same file on another machine (normally, I’m on a Mac, this upload was from Windows) - same effect
3. did a completely different edit without the island data I want to upload (affecting some borders and cities all over Nor∂urland - no post processing delay!
4. followed your advice to check if the data is on the server. While uploading some islands again, when the “postprocessing” message appeared, I looked into the “History” tab on the OGF website and inspected the changeset. All nodes and polygons were listed there.
5. taking 4 one step further, cancelling the upload when JOSM goes into the post processing phase - seems that all objects have been uploaded. Tried this several times.
Comment from Luciano on 17 June 2018 at 18:54
Great troubleshooting. And interesting result. So the problem is in JOSM. How much RAM do you have?
Here’s a thought: on large changesets, JOSM goes to do its normal “housecleaning” after the upload, and because the “before” and “after” versions of the file don’t simultaneously fit in your system’s memory at the same time, it does some kludgey thing where it “switches” the two files in and out of memory, and somehow some coder messed up and it goes back and forth for each object, under some certain set of conditions. This would produce the kinds of exponential delays we’re looking at. Call it a JOSM problem. In general, in my past experience, Java apps are bad at memory management - so it might even be a Java problem (the code platform JOSM is built on).
What the JOSM coder might say: get more memory on your machine! Problem solved!
My suggestion, to avoid the problem: 1) reduce the size of your uploads - subdivide the work, put some of the work to be uploaded into a new file, upload the new file (while closing the other), repeat; 2) wait for someone smart at JOSM to fix the problem eventually; 3) meanwhile, look into getting a bigger computer (more RAM memory).
Comment from EMKLI on 17 June 2018 at 19:39
Gooooooooooodness, I feel so ridiculous now… I found the “error” - by accident. Long story short: from the OSM file, I did selective uploads, e.g. drawing a selection rectangle over the objects - selecting nodes AND ways. THIS was the problem. Now I’ve found out that when I ONLY select the paths and NOT the nodes, the post processing doesn’t occur. I’ve just uploaded 35.000 objects in less than 5 minutes. Big big facepalm. :D
So, a message to all: beware when you perform selective uploads from within JOSM. Try to unselect all nodes which are part of a path/polygon.
Comment from Luciano on 17 June 2018 at 19:51
Good information. It somewhat negates my speculation about memory management problems, although really it just transfers the blame to the coder in JOSM responsible for that selective upload feature. Anyway, glad you found the problem. I have hardly ever used the selective upload feature, that I remember. Perhaps I tried it at some point when I first started using it, and saw the problem you experienced, and that’s why I stopped using it.
Comment from EMKLI on 17 June 2018 at 20:00
Well, I have 42 Gig of RAM with 16 Gig allocated to the JVM, thanks to Paxtar who responded to my last issue with the extremely slow JOSM UI. I guess that’s not the problem. ;) It really seems that I have to file a bug report over at the JOSM repository.
Comment from Luciano on 17 June 2018 at 20:35
Wow. Then memory per se is definitely NOT the cause of the problem. I limp along with a junky 8 year old Korean PC clone running Linux and only 4 GB RAM, and rarely have problems if I keep my .OSM files under about 70 MB. I agree you should file some kind of bug report (or look for existing bug) regarding the selective upload feature.