Cleanup

[Kafuuka]

I think it is best to split this from the main discussion about MDscripting.
[quote]Kafuuka, you raise a very important point. The cleanup is sorely lacking for both keys and storage. Yes, the empty storage removes itself but things like "tests" with data in them stay behind, I think a storage list which shows the names of your storages would be useful (even if you just get them in an array)[/quote]

The problem is simple. Every time a new variable or key is created, it increases the server load a tinsy bit. Memory/harddisk space and possibly processor time are eaten. cpu time? Imagine a query on all keys a player has. I have only one reason I can think of to make such a query, but that is one too many.
Quests can be forgotten by players and creators alike, but the server will only delete the keys if the creator asks it. This is also true for scripts themselves.

Ideally all creators are excellent, motivated programmers and take the effort necessary to minimize this. In practice most of us are liable to be happy if our scripts work regardless of efficiency. There a couple of guidelines which are valid for every programming language that has to deal with memory:
-For every line of code that creates a new variable, there should be a line of code to delete that variable.
-Use good names for your variables. Compare 'x143' and 'number_of_attempts'.

Both of these guidelines are problematic. Only keys can be named. Quests are often one time only, thus requiring an end key to be stored as long as the quest exists. This implies that cleaning up quest scripts can only be done if you also clean the keys which then become redundant. If your script is spread over a dozen clickies, this isn't fun work and actually a process which can be standardized:
if scripts can be labeled, then it becomes a straightforward idea to delete all scripts with the same tag. Deleting the endkeys (and others if necessary) requires a query over all players but is also programmable. Both of these are not doable in MD script afaik, but should be implemented asap to counter bloated code. In one step I would force labeling to be mandatory for scripts and keys. It is more work for Mur now, and more work every time you make a new key/script, but the maintenance will go down a lot.

[I am Bored]

perhaps you could echo the variables? here is a script that you can put in your code, it will echo all of your currently set variables, they are labled as settings because variable isn't an allowed word, if you want i can post my one that echo's the user vars as well, but that is currently posted in the manual, this script now echos all settable variables.
[code]echo "setting a: ".@va." <br />";
echo "setting b: ".@vb." <br />";
echo "setting c: ".@vc." <br />";
echo "setting d: ".@vd." <br />";
echo "setting e: ".@ve." <br />";
echo "setting f: ".@vf." <br />";
echo "setting g: ".@vg." <br />";
echo "setting h: ".@vh." <br />";
echo "setting i: ".@vi." <br />";
echo "setting j: ".@vj." <br />";
echo "setting k: ".@vk." <br />";
echo "setting l: ".@vl." <br />";
echo "setting m: ".@vm." <br />";
echo "setting n: ".@vn." <br />";
echo "setting o: ".@vo." <br />";
echo "setting p: ".@vp." <br />";
echo "setting q: ".@vq." <br />";
echo "setting r: ".@vr." <br />";
echo "setting s: ".@vs." <br />";
echo "setting t: ".@vt." <br />";
echo "setting u: ".@vu." <br />";
echo "setting v: ".@vv." <br />";
echo "setting w: ".@vw." <br />";
echo "setting x: ".@vx." <br />";
echo "setting y: ".@vy." <br />";
echo "setting z, usually credit to creator: ".@vz." <br />";
echo "template replacement setting: ".@tpl." <br />";
echo "log setting: ".@log." <br />";
echo "temporary setting: ".@temp." <br />";
echo "input setting: ".@input." <br />";
echo "content: ".@content." <br />";
echo "storage: ".@storage." <br />";[/code]

Edited October 31, 2009 by I am Bored

[Observer]

Except that every script has it's own variables... So you'd only echo the variables of that one script of your own, right? (and maybe those in your own array)

[I am Bored]

that's why i said it had to be put in your script.when the databases work i will make another to echo those.

Edited October 31, 2009 by I am Bored

[Kafuuka]

You are making the assumption people will clean up things themselves. Perhaps all those who currently have access have the intention to do so and the skill, but at some point there will be people lacking intent and/or skill. Besides the road to hell is paved with good intentions.

[I am Bored]

yes but it currently takes a pretty large effort to get enough wishpoint to acess it, i have been playing for 1&1/2 years, and i only have 7 wishpoints

[Rendril]

So Kafuuka, what you are proposing is to "label" the scripts which belong to a certain quest which allows you to delete them in one call?
Storages would also need ot be given a label which corresponds to the script's label, so that the garbage collector knows to take it down too.
The problem with keys is that they are not editor-specific, in other words 2 people could be using a key of the same name (that is why I suggest using key prefixes in all your code) how would the cleanup know which ones to take?


IAB, if you are using echo to see what is inside the variable, rather use debug()
It works incredibly well, especially on arrays.

[I am Bored]

whats debug? all i am doing is what i know how to do.

[Rendril]

Have a look at the manual for [url="http://magicduel.invisionzone.com/topic/5497-debugging-you-code/"]debugging your code[/url]

[Kafuuka]

[quote name='Rendril' date='31 October 2009 - 08:07 PM' timestamp='1257016079' post='46227']
So Kafuuka, what you are proposing is to "label" the scripts which belong to a certain quest which allows you to delete them in one call?
Storages would also need ot be given a label which corresponds to the script's label, so that the garbage collector knows to take it down too.
The problem with keys is that they are not editor-specific, in other words 2 people could be using a key of the same name (that is why I suggest using key prefixes in all your code) how would the cleanup know which ones to take?[/quote]
I propose there to be some structure enforced. All scripts should have a label and there should be a list of keys that exist. As soon as one player has a key, or even is capable of receiving a key, that key 'exists'. Afaik this list is currently not implemented and without it there will be useless keys piling up without anyone realizing they exist, which is like a memory leak. To prevent this, it makes sense to assign key names to the label of your quest scripts too. At the time of writing a script, you know which keys you create and you can list them. When deleting all scripts with a certain label, a query can be executed on all players, to remove the keys in that list. (which is something you wouldn't want to do every second i guess, but then again who is going to delete many scripts in a short time?) This relies on people accurately listing the keys. Alternatively you can force people to register keys with a label before they can be given to a player. (Could be verified at the time the script is tested for security.)

How it works:
1. create a label L
2. register keys K to L
3. create scripts S with label L
Steps need to be carried out in order, but you don't need to do all of them. eg. no scripts, only a label.

When deleting L, first all scripts S will be deleted, if any. Then all keys K will be removed from all players that have them, if any. Finally the label L is deleted.

If you have cross-quest keys, you can put them in a different label. You could even put each key in a different label, thereby making it redundant. However using labels in a sensible way (ie. making as few labels as possible), you needn't worry too much about cleaning. In [post='46174']this post[/post] I outlined an example of a quest that should leave only one key on players at the end. However, if a player stops midways, they will have multiple keys. If at a later time you want to purge all remnants of the quest, it is very easy to forget the additional keys of players who gave up on the quest and only clean the end key. (afaik We can't use the required queries in MDscript anyway.)

Two large issues: people using the labels wisely and people actually pushing delete once in a while. My solution thus is far from perfect, but at the moment there is no cleaning at all. I hope it can be improved upon so that we will not need code inspectors. There is already a fair bit of paranoia about plagiarism and cheating, code inspection would not help that at all. Reading undocumented code is annoying anyway.


Storages: I'm confused how they work, so I'll get back to those once I figure them out.

[Observer]

What about having different kind of keys/scripts that "expire" at a certain point? Keys that'll be used for a requirement to future quest (end keys) could be unlimited but if you know your quests is only going to be up for a month you could use a "1 month" key. As for scripts, they should "all" expire sooner or later as there isn't much point in leaving one around forever.


There'd be a clean-up at the end of every month or every 14 days as not to use CPU/memory for this fulltime

Also, maybe it could be connected to the account cleanup, so that when a player's account is archived the permanent keys set by him/her would be removed.

I'm not sure if this is "easily" codable and how much effort it'd take... But it pretty much eliminates the possibility of forgetting about your key.

[No one]

[quote name='Observer' date='01 November 2009 - 12:43 AM' timestamp='1257029003' post='46237']
What about having different kind of keys/scripts that "expire" at a certain point? Keys that'll be used for a requirement to future quest (end keys) could be unlimited but if you know your quests is only going to be up for a month you could use a "1 month" key. As for scripts, they should "all" expire sooner or later as there isn't much point in leaving one around forever.


There'd be a clean-up at the end of every month or every 14 days as not to use CPU/memory for this fulltime

Also, maybe it could be connected to the account cleanup, so that when a player's account is archived the permanent keys set by him/her would be removed.

I'm not sure if this is "easily" codable and how much effort it'd take... But it pretty much eliminates the possibility of forgetting about your key.
[/quote]

If there could be a "garbage collector" kind of thing would be nice, but ... keys ... MUST be kept for ever.
Also, quests are not the only thing that can be coded with MDScript, features can be created too.

So, yes, there is a need to purge keys, but there must be a way to control it.

Also, do rememeber that from same script for same thing there can be 2 different keys for 2 different players: KEY_OWNER_PLAYER_name_of_key.

So, what I propose to you is to consider throwing here some ideas on how to count all these keys and how to purge them. I think that Mur should be involved in this as he knows the code behind the keys and he may have an easier solution for all this.


So, my suggestion is to have somewhere a list of keys that you as creator gave. If there are more keys with same name given to different players, they should be grouped toghether.

[Kafuuka]

@Observer:
not all scripts should expire. eg. Someone could make a market script to finally get rid of all those willing to trade topics on the forum. That script should only expire if a better market script becomes available and I doubt that would happen at a predictable time. In general it is difficult to predict when a key/script will expire, unless you set a deadline for your quest and do not extend it... Imagine the server going down for maintenance at the last day, at such a time i would extend the quest 24h.

@No One:
Labels work similar to the list you mention. Makes me wonder if you read my post
And yes Mur should be involved with this, especially since it requires changing the MD code, not writing MD scripts. However it would be nice to give him a list of solutions. Or we could brainstorm together on irc, since this is a problem that deserves priority.

[I am Bored]

perhaps having an extra variable that's default setting is to clear that script out every 2 months, but is settable to never clear, so that things like my book, could always be accessable without the keys being reset, and you having to do the quest again.

[Rendril]

The keys are stored an a serialized array, the structure cannot be changed easily.
It affects every location you can access.

As I said, there is no "creator" stored for a key. While a good idea it is not an easy task ot implement. You would also be charging an additional overhead on the server of now storing 2 values for every key, effectively doubling the key-use anyway.

Maybe a simple solution is to append the editor ID and if desired, an expiration date for a key.

Edited November 1, 2009 by Rendril

[Kafuuka]

Keeping a list of keynames changes nothing to the keys themselves. If we assume the average key is possessed by ten players, using the list implies one more string constant is saved, increasing the load by 10% if a key is indeed only a string (I think I read the date is saved too, which implies the effective increase is less than 10%).
If one in five keys (quests) expires within two months and the total number of keys is more or less a constant, then we can estimate after one year as:
without cleanup: 80 permanent keynames + 6*20 temporary keynames = 200 keynames; loading = 10 => total memory used 2000
with cleanup: 80 permanent keynames + 20 temporary keynames = 100 keynames; loading = 11 => 1100
The values are guesses, but a key that is possessed by less than 10 players should not be the norm. Whether features or quests, the idea is that most of them are made for the masses. If a quest expires, two months is more than we usually give. At the moment most quests expire too, but given the script it will be more feasible to make permanent quests because there is less maintenance on them.

Things people seem to forget:
- keys should expire at the same time all scripts giving/taking the keys expire.
- deciding which keys/scripts to clean up at what time is not the same as deciding how to clean them up. Without the how, the what and when is irrelevant.
- the chances people set the expire date to never are only slightly smaller than the chances people forget to push delete after the expiration date. In general it are the same people that do not care about cleaning up or are sloppy and forgetful, that are either to vain or to neglecting to think ahead and decide a good expiration time.

[I am Bored]

[quote name='Rendril' date='01 November 2009 - 03:20 PM' timestamp='1257110456' post='46333']
The keys are stored an a serialized array, the structure cannot be changed easily.
It affects every location you can access.

As I said, there is no "creator" stored for a key. While a good idea it is not an easy task ot implement. You would also be charging an additional overhead on the server of now storing 2 values for every key, effectively doubling the key-use anyway.

Maybe a simple solution is to append the editor ID and if desired, an expiration date for a key.
[/quote]


[quote name='Kafuuka' date='01 November 2009 - 04:25 PM' timestamp='1257114302' post='46338']
Keeping a list of keynames changes nothing to the keys themselves. If we assume the average key is possessed by ten players, using the list implies one more string constant is saved, increasing the load by 10% if a key is indeed only a string (I think I read the date is saved too, which implies the effective increase is less than 10%).
If one in five keys (quests) expires within two months and the total number of keys is more or less a constant, then we can estimate after one year as:
without cleanup: 80 permanent keynames + 6*20 temporary keynames = 200 keynames; loading = 10 => total memory used 2000
with cleanup: 80 permanent keynames + 20 temporary keynames = 100 keynames; loading = 11 => 1100
The values are guesses, but a key that is possessed by less than 10 players should not be the norm. Whether features or quests, the idea is that most of them are made for the masses. If a quest expires, two months is more than we usually give. At the moment most quests expire too, but given the script it will be more feasible to make permanent quests because there is less maintenance on them.

Things people seem to forget:
- keys should expire at the same time all scripts giving/taking the keys expire.
- deciding which keys/scripts to clean up at what time is not the same as deciding how to clean them up. Without the how, the what and when is irrelevant.
- the chances people set the expire date to never are only slightly smaller than the chances people forget to push delete after the expiration date. In general it are the same people that do not care about cleaning up or are sloppy and forgetful, that are either to vain or to neglecting to think ahead and decide a good expiration time.
[/quote]

But you are both thinking about 2 different things, rendril, you are talking about the amount of stuff the server is currently doing, while you kafuuka are talking about stuff that is stored on the server, but the final question is, is the exchange worth it? in the end i would say yes, because it will save space, and we currently there isn't much strain on the server, and i know that there currently isn't a big strain on the server, because mur himself has said that there isn't a big strain on the server.

[Rendril]

I'm not sure what you mean. I was talking about the server having to store more as well.
It is not a heavy strain, no. But an unnecessary one.

[Kafuuka]

[quote name='Rendril' date='02 November 2009 - 12:48 AM' timestamp='1257119300' post='46344']
I'm not sure what you mean. I was talking about the server having to store more as well.
It is not a heavy strain, no. But an unnecessary one.[/quote]
Imo it is nothing short of a memory leak. The person who creates a key giving script cannot make a script that will take away said key with a 100% efficiency. It is a small leak, perhaps only a kilobyte a month... yet no programmer should be satisfied with that.

[Rendril]

I agree, it is not satifactory.
But the solutions proposed are as expensive as the extra keys, if not moreso.

[Kafuuka]

Care to explain how you estimate the expenses?
My rough estimate for the label method gives a profit ~ 50% after one year. If you combine it with the solution in the other thread [post='46394'](delete button)[/post], it would break even after a year and yield 30% profit after two years. The parameters I chose for the estimate are not exaggerated in favor of my method at all. And I did not take into account that string constants often use more memory than integer constants.

[Rendril]

Storing an integer identifer is not the problem (well, it's a rather small one comparitively)
The problem in my view coems from trying to select keys held by other plqyers, not the one accessing the item.

I don't know MySQL's performance ranges on the tables but I am fairly certain that scanning 30k+ records, multiple times, will not be cheap.

[Kafuuka]

Those queries should only occur when one deletes a quest (at least in this system, what the other thread is aiming at is interesting too... jolly good fun with dual threads, which I guess is my fault.)

[Rendril]

I somehow didn't see the post you made about the key storing over time.
A script could be labelled so it knows to be deleted, but you would need to assign which keys belong ot it specifically.
In some cases I want keys to be around for ages, some I want gone within days. The direct deletion trigger or expiration would be very useful, but how do we link them?

At least we adhere to mutual exclusion when dual-threading
[color="#FFFFFF"]Those of us who get this joke need help[/color]

[Kafuuka]

[quote name='Rendril' date='02 November 2009 - 07:12 PM' timestamp='1257185539' post='46421']
I somehow didn't see the post you made about the key storing over time.
A script could be labelled so it knows to be deleted, but you would need to assign which keys belong ot it specifically.
In some cases I want keys to be around for ages, some I want gone within days. The direct deletion trigger or expiration would be very useful, but how do we link them?[/quote]
[quote name='Kafuuka' date='31 October 2009 - 11:22 PM' timestamp='1257027755' post='46236']
How it works:
1. create a label L
2. register keys K to L
3. create scripts S with label L
Steps need to be carried out in order, but you don't need to do all of them. eg. no scripts, only a label.

When deleting L, first all scripts S will be deleted, if any. Then all keys K will be removed from all players that have them, if any. Finally the label L is deleted.[/quote]
And if you want a timer, you should add it to L.
It is possible to enforce keys to be assigned to a label. Whenever the security check reviews a new script, it can search for the commands involving keys, eg. mds_has_rpcq_keys(key1, key2). It can extract the names key1 and key2 and check if those keynames exist for any label or for the label to which your script is assigned. The latter provides more security, but then you need to allow a label to be assigned to another label, if you want to use keys from that other label.

One more possible flaw: if a script is deleted, but a person opened the clicky containing it before the deletion and pushes a button afterwards, that script is probably not stopped? This could be circumvented by delaying the deletion of keys until say 24h after the scripts are deleted. Not overly difficult yet quite an annoying thing.