LUA: add values in nested table

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

LUA: add values in nested table

Wilmar Pérez
Hi all,

I have a performance issue in my application. I would like to gather some ideas on what I can do to improve it. The application is very easy: I need to add values inside a nested table to get the total an user wants to pay out of all the pending payments. The user chooses a number of payments and I calculate how much it is they will pay.

This is what I have:

jsonstr = "{ "name": "John", "surname": "Doe", "pending_payments": [ { "month": "january", "amount": 50, }, { "month": "february", "amount": 40, }, { "month": "march", "amount": 45, }, ] }" local lunajson = require 'lunajson' local t = lunajson.decode(jsonstr) local limit -- I get this from the user local total = 0; for i=1, limit, 1 do total = total + t.pending_payments[i].amount; end;







It works. At the end I get what I need. However, I notice that it takes ages to do the calculation. Each JSON has only twelve pending payments (one per month). It is taking between two to three seconds to come up with a result!. I tried on different machines and LUA 5.1, 5.2., 5.3. and the result is the same.

Can anyone please suggest how I can implement this better?

Thank you!

Wilmar

PD: I posted the same question on stackoverflow.  


Reply | Threaded
Open this post in threaded view
|

Re: LUA: add values in nested table

Francisco Olarte
Wilmar:

On Fri, Jun 12, 2020 at 6:14 PM Wilmar Pérez <[hidden email]> wrote:
> I have a performance issue in my application. I would like to gather some ideas on what I can do to improve it. The application is very easy: I need to add values inside a nested table to get the total an user wants to pay out of all the pending payments. The user chooses a number of payments and I calculate how much it is they will pay.
...
> It works. At the end I get what I need. However, I notice that it takes ages to do the calculation. Each JSON has only twelve pending payments (one per month). It is taking between two to three seconds to come up with a result!. I tried on different machines and LUA 5.1, 5.2., 5.3. and the result is the same.

As people in SO told you, this is not lua code, try posting some real example.

I very much doubt the table manipulation is taking that much time, I
do much heavier ops in microseconds in the code I'm working right now.
I suspect you are not timing what you are giving us.You may be getting
a HUGE value for limit, although I suspect this will end in error.

I've made some timings with this little test:
$ cat tstspeed.lua
local base=os.time()
local function P(...)
   local t = os.time()
   print(t, t-base, ...)
   base=t
end

local limit = 100000

P("Starting, filling table with "..limit.." rows.");

local p = {}
for i=1,limit do
   p[i]={ month = "month"..i, amount=i }
end
local t = { name ="John",surname="Doe",pending_payments=p }

-- straightforward:
local repeats = 10000
P("Starting straightforward loop.","repeats ="..repeats)
for r=1,repeats do
   local total = 0;
   for i=1, limit, 1 do -- last 1 is not needed anyway.
      total = total + t.pending_payments[i].amount;
   end
end
P("Starting local payments loop.","repeats ="..repeats)
for r=1,repeats do
   local total = 0;
   local pending_payments = t.pending_payments
   for i=1, limit, 1 do -- last 1 is not needed anyway.
      total = total + pending_payments[i].amount;
   end
end

P("Terminated.")

With the origial code I get this times ( note I do a 100k table ad 10k repeats )

folarte@7of9:~/tmp$ lua tstspeed.lua
1591980496    0    Starting, filling table with 100000 rows.
1591980496    0    Starting straightforward loop.    repeats =10000
1591980515    19    Starting local payments loop.    repeats =10000
1591980532    17    Terminated.

So this is aprox. 1.9 ms for a 100k table with 10k repeats, caching
payments saves a little, as expected.

If I turn to a hundred element table and a million iterations...

1591980752    0    Starting, filling table with 100 rows.
1591980752    0    Starting straightforward loop.    repeats =1000000
1591980754    2    Starting local payments loop.    repeats =1000000
1591980755    1    Terminated.

It seems os.time lacks precision for 'only' a million iterations. So
let's try a thousand elements.

1591980773    0    Starting, filling table with 1000 rows.
1591980773    0    Starting straightforward loop.    repeats =1000000
1591980791    18    Starting local payments loop.    repeats =1000000
1591980804    13    Terminated.

so 1.8/1.3 microseconds per loop for 1k elements. Seems pretty fast.
So either your problem is in lunajson or in something you are not
showing or my machine is much faster, which I doubt.

> Can anyone please suggest how I can implement this better?

Measure what is slowing you down really, make a small reproducible
test case if you have it. Post it here.

btw, Debian 10, lua 5.3.3, AMD Ryzen 5 3600X not overclocked, not too fast.

Francisco Olarte.
Reply | Threaded
Open this post in threaded view
|

Re: LUA: add values in nested table

Wilmar Pérez
Thank you Francisco.

You were absolutely right, the limit was the problem. I found the delay had nothing to do with the calculation in LUA. It was related with a configurable delay in the retrieval of the limit variable.


Thanks very much for your reply!
Reply | Threaded
Open this post in threaded view
|

Re: LUA: add values in nested table

Philippe Verdy-2
you can eventually test:
    if (t.pending_payments[i] == nil) then break end
inside your loop (if you assume that the list of months has no "hole", but you don't want to set a limit for your for loop.

Anyway the for loop should better be using ipairs() so that you can still accept "holes" in months:

  for i,payment in in ipairs(t) do
    if limit and i > limit then break end
    total = total + payment.amount;
  end

(your sample only shows named months probably of the same year, but the actual data may be an arbitrary month in ISO format "yyyy-mm" or using a second property for the year. In which case the "limit" should be able to compute if the date match the limit which could accept either a maximum number of months over possibly several years, or a date limit also in "yyyy-mm" format).
Also nothing indicates that the JSON data is necessary ordered: eahc payment has its own month, they may occur in arbitrary order, or there would be several payments for the same month in different rows...
So check your data model for your JSON source!

Le ven. 12 juin 2020 à 19:17, <[hidden email]> a écrit :
Thank you Francisco.

You were absolutely right, the limit was the problem. I found the delay had nothing to do with the calculation in LUA. It was related with a configurable delay in the retrieval of the limit variable.


Thanks very much for your reply!
Reply | Threaded
Open this post in threaded view
|

Re: LUA: add values in nested table

Wilmar Pérez
Hi Philipe,

Thank you very much. This is very relevant. My JSON is ordered and then month actually comes as "mm" only but I think I will implement your suggestion to improve the code resiliency to changes in the JSON feed.

Best,

Wilmar

Le ven. 12 juin 2020 à 17:57, Philippe Verdy <[hidden email]> a écrit :
you can eventually test:
    if (t.pending_payments[i] == nil) then break end
inside your loop (if you assume that the list of months has no "hole", but you don't want to set a limit for your for loop.

Anyway the for loop should better be using ipairs() so that you can still accept "holes" in months:

  for i,payment in in ipairs(t) do
    if limit and i > limit then break end
    total = total + payment.amount;
  end

(your sample only shows named months probably of the same year, but the actual data may be an arbitrary month in ISO format "yyyy-mm" or using a second property for the year. In which case the "limit" should be able to compute if the date match the limit which could accept either a maximum number of months over possibly several years, or a date limit also in "yyyy-mm" format).
Also nothing indicates that the JSON data is necessary ordered: eahc payment has its own month, they may occur in arbitrary order, or there would be several payments for the same month in different rows...
So check your data model for your JSON source!

Le ven. 12 juin 2020 à 19:17, <[hidden email]> a écrit :
Thank you Francisco.

You were absolutely right, the limit was the problem. I found the delay had nothing to do with the calculation in LUA. It was related with a configurable delay in the retrieval of the limit variable.


Thanks very much for your reply!


--
--------------------------------------------------------
Wilmar Pérez