Can someone tell me what going wrong here?

Jip

This is because of floating point imprecision. Note that the difference is not extreme - it is actually really small. It uses the scientific notation.

If you want it to not go past three decimal points then you can multiply it by a 100, floor it and then multiply it with 0.01.

Resin_Smoker

@jip well the problem I'm seeing is the exponent, isn't being seen as such when a direct comparison is made. In this instance the script is comparing the units current location vs the surface height. Should the units elevation be lower than the surface the unit is warped to the surface.

If you look and those end values of the above WARN outputs, the exponents are pretty small values, but show up as whole numbers.

Rather than -6.103515625e-005, the script sees "-6". Causing the unit the warp over and over again.

What's needed here is a way to limit the results to just 2 or 3 decimal places.

Resin_Smoker

I may give this a try..https://stackoverflow.com/questions/57950030/wow-rounding-to-two-decimal-places-in-lua

This is also good: https://wiki.multitheftauto.com/wiki/Math.round

Jip

You don't need to convert it to a string. If you provide a bit more context then I may be able to help you more

Resin_Smoker

This post is deleted!

Resin_Smoker

@jip

For a given number such as... 19.123456

I need to cut off the last digits specified. Hence with 3 digit precision, the output should be "19.123". Wanting to be able to feed in the decimal place and get only that, no zeros trailing afterwards.

Resin_Smoker

Finally after much research and experimentation, this (sorta) works...

function GetTruncatedNumber(number, precision)
   local fmtStr = string.format('%%0.%sf',precision)
   number = string.format(fmtStr,number)
   return number
end

Output...

Input 18.4453125 
Output one decimal:   18.4 
Output two decimal:   18.45
Output three decimal:   18.445

Jip

@jip said in Can someone tell me what going wrong here?:

If you want it to not go past three decimal points then you can multiply it by a 100, floor it and then multiply it with 0.01.

I feel like I've already mentioned how to do that, have you tried this?

Resin_Smoker

@jip I'm trying to create a function where all that i need to add is the number and the decimal places wanted. I'm close, yet the above function does do a small bit of rounding.

Input 18.3671875	       
one:	18.4	        
two: 	18.37 (Rounded!!!)	    
three: 	18.367

The functions where math.floor and ceil are used can produce some rather wonky results. Which is why I'm experimenting.

Resin_Smoker

@jip

Tried your method...

function GetTruncatedNumber02(num, dp)
	local pwr = math.pow(10, dp)
	local inverse = math.pow(10, -dp)
	WARN('	pwr: ', pwr,'	inverse: ',inverse)
	return math.floor(num * pwr) * inverse
end

Results

WARNING:        Input to  math Truncate:	18.404296875
	    	    	    
WARNING:         pwr: 	10	 inverse: 	0.10000000149012
Output one:	18.39999961853

WARNING:         pwr: 	100	  inverse: 	0.0099999997764826
Output two: 	18.39999961853

WARNING:         pwr: 	1000	   inverse: 	0.0010000000474975
Output three: 	18.404001235962

Still no dice.... Why is it so bloody hard to truncate a number in lua?

Sheikah

It is hard to truncate a number because all numbers in lua are represented as double floating pointvalues which will have the precision issues you are seeing. If you are just looking to see if two numbers are the same why not add in some tolerance? so just a-b<epsilon where epsilon is some value like 1e-4

Resin_Smoker

@sheikah

After hours of tinkering... Success!

function GetTruncatedNumber02(num, dp)
   local pwr = math.pow(10, dp)
   num = math.floor(num * pwr)
   WARN('	pwr: ', pwr,'	num', num,'	num / pwr: ', num / pwr)
   local result = num / pwr
	
   local fmtStr = string.format('%%0.%sf',dp)
   number = string.format(fmtStr, result)
   return number
end

Results

Math Truncate:	18.3671875    Decimals given, 1,  2,  3

Decimal: 1         pwr: 	10	 num	183	  num / pwr: 	18.299999237061
Output one after string filter:	18.3

Decimal: 2         pwr: 	100	  num	1836	     num / pwr: 	18.360000610352
Output two after string filter: 	18.36

Decimal: 3         pwr: 	1000	   num	18367	        num / pwr: 	18.367000579834
Output three after string filter: 	18.367

Ended up combining a few methods to get this result. Not sure how it would behave with negative values but thats for tomorrow. (tired)

Resin