Parallelization av import skriptet

[Thomas Sødring]

On 8/23/21 3:55 PM, Gareth Western wrote:
>
> I’m making some progress with testing the import of this large 
> uttrekk. The next issue I’d like to look at is performance. Serial 
> upload is too slow when the archive contains hundreds of thousands of 
> documents. A quick analysis of the structure indicates that there are 
> approximately 400,000 documents and 100,000 folders (mappe). Would it 
> be safe to do at this level? Or should I try at a higher level first 
> (e.g. klasse, of which there are approximately 4000)?
>
>
Great news! We really appreciate the real world testing! nikita should 
not have any scalability issues with the database. The numbers you note 
is not that much so I hope nikita handles it nicely. It certainly makes 
sense to first try the import of 4000 klasse. I assume you are not 
importing anything underneath that (e.g. saksmappe). Such an import 
should be quick.

If you have various arkivdel in your extraction you can try to limit the 
import at the arkivdel level. Or you could try to limit it at the klasse 
level just to get any idea of performance.

We would be very happy to see performance results here on the list. Good 
or bad :)

  - Thomas



-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.nuug.no/pipermail/nikita-noark/attachments/20210823/472eff6f/attachment.htm