I am developing a web application based on Spring 4 which aims to act as some kind of gallery for hosting images. Therefore, I have to implement a Java client application that takes care of uploading the images to the web application because of a huge number of images (can be up to 100000 images) that have to be uploaded by the client.
I have currently implemented a REST endpoint web application which is able to receive one image after the other and save it on the server side. For each image to upload, a client application makes a POST request to the REST endpoint containing the image. Considering the fact that these images should be available on the server web application as soon as possible, I guess that this is not the optimal solution for this job. I think that this solution doesn't even use the full bandwidth which would be available.
Now I am asking how to implement this feature in a reasonable and efficient way that makes use of the full available bandwidth (even possible without REST)?
There are various ways this can be done but just looking at the RESTful server way which you have started then my suggestion is to upload the files in parallel.
Would it be possible for your client to have multiple worker threads. The worker threads read off a shared blocking queue. You decide how many worker threads to create. The main thread will figure out which files need to get uploaded. The main thread will then enter into the queue the file location (either relative or full path or URL) into the queue. The worker-threads will each grab a file upload request from the queue and will make the POST request to upload the file.
This would allow you to make better use of the bandwidth. You can then add smarts to decide how many worker threads to use. Maybe by polling the REST server to ask it how many worker threads it should use. The server could have an AtomicInteger counter that has number of current uploads. It can return MAX_UPLOADS - currentUploads.get().
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments