Benchmarking the insertion of 8333 post journal entries

Thomas Sødring Thomas.Sodring at hioa.no
Tue May 16 14:30:45 CEST 2017


Hi,


Good stuff! Let's be honest, that's not good reading.


The amount of data being stored is very small so 41 KiB per entry is strange. If you are using H2 (in memory db) then I understand that there is a

cost as indexes etc are being created. But we should also test on top of mysql/postgres. I'm not sure what you have persisted the data to.


There are 3 things that should be considered before benchmarking.


  1.  Database
  2.  In demo mode, we run in log DEBUG
  3.  Hibernate is printing all SQL commands


To change 2.

https://github.com/HiOA-ABI/nikita-noark5-core/blob/master/core-webapp/src/main/resources/application-demo.yml#L13


should be set INFO or WARN


To change 3.

https://github.com/HiOA-ABI/nikita-noark5-core/blob/master/core-webapp/src/main/resources/application-demo.yml#L38


Set this to false.


This is not going cause a major increase in processing time, but it's something.


When it comes to 1. I believe there are a few indexes in place so that might explain the memory usage. We just need to run on mysql to check.


Maybe I should add a description to get mysql integration working, it should work out of the box but it's been so long since I  looked at the code.


 - Tom



________________________________
From: nikita-noark-bounces at nuug.no <nikita-noark-bounces at nuug.no> on behalf of Petter Reinholdtsen <pere at hungry.com>
Sent: Saturday, May 13, 2017 6:04:39 PM
To: nikita-noark at nuug.no
Subject: Benchmarking the insertion of 8333 post journal entries


Just for fun, I did a small benchmark of the Nikita Noark 5 Core API
running on my laptop (machine is slighly short on memory), to see of
much time the sqlite2noark5core script would use to insert 8333 into the
database via the API.

This took 64 minutes and 24 seconds.  According to top, the java process
started with sizes virt 5580736, res 819272 and shr 23052.  When the job
was done, the java process had sizes virt 5604600 res 1.109g and shr 0.

This mean the script was able to insert around 2.2 entries per second,
and the java process resident size grew 343599 KiB, ie 41 KiB per entry.

For each entry the script create a 'saksmappe' entry, a 'journalpost'
entry, zero or more 'korrespondansepart' entries and finally one
'dokumentbeskrivelse' entry.  At the start it creates one 'arkiv' and
'arkivdel' entry.

Thought it best to mention the numbers here to have some kind of
baseline when we start to look at profiling the API to speed it up. :)

--
Happy hacking
Petter Reinholtsen
_______________________________________________
nikita-noark mailing list
nikita-noark at nuug.no
https://lists.nuug.no/mailman/listinfo/nikita-noark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.nuug.no/pipermail/nikita-noark/attachments/20170516/247614a4/attachment.htm 


More information about the nikita-noark mailing list