Sunday, August 4, 2013

User's Guide to Fold.it Lua 2.0 API

The site, http://fold.it, is a site where you can 'fold for science'. Fold.it is a program, a client, that allows scientists and non-scientists to play a game of folding proteins.

The problem of folding proteins is an inherently geometrically hard problem.

People have been found to have intuitive ideas on what the shape of the protein should look like. The game Fold.it was created to allow people to express this intuition.

The 3D shape of the protein is the most important property of a protein. The shape allows scientists to determine how the protein functions.

So how is all this something for a computer science blog?

Fold.It introduced a scripting language to allow repetitive folding tasks to be automated. There is a GUI scripting tool and a Lua scripting tool.

The Lua language allowed with Fold.it is a subset of the official langue. http://www.lua.org/ with an additional API for folding and client functions.

At Fold.it, a Lua script is called a 'recipe'.

Below is a user's guide for the Fold.it Lua Version 2.0. There was a version 1.0. It has been deprecated. Old version 1.0 scripts can still be run, but one cannot create new 1.0 scripts or modify 1.0 scripts without upgrading to version 2.0 API.

See the following two Lua recipes for the mapping of Lua 1.0 to Lua 2.0 functions. You will need to be running the Fold.it client in order to get a human readable version of the recipes.
http://fold.it/portal/recipe/43867
http://fold.it/portal/recipe/31012

Fold.it Lua 2.0 API
Lua V2.0 is the fold.it 'programming' language. It is a subset of the Lua 5.0 language. http://lua.org with an additional API.

The standard Lua 5.0 functions supported in Fold.it Lua V2.0 are found at http://foldit.wikia.com/wiki/Lua_Standard_Libraries.

The reader is expected to be familiar with how to run the Fold.it GUI client. Terminology here is the same as that found in the GUI.

The reader is also expected to have an understanding of programming and the syntax of Lua. If you are new to programming or Lua, please read here first, and elsewhere. http://foldit.wikia.com/wiki/Lua_Scripting

Programming Note: One does not have the ability to use libraries, other than those in the Lua standard libraries, or refer other files in a Fold.it Lua V2.0 script. A Fold.it Lua script is one large file. Thus it is a good habit to keep a file that contains all the custom functions you have written and will need from script to script and simply paste them into the top of each new script you write.

Short Example

When using the Fold.it client, one can hover over a segment and press <tab>.
This brings up a note.

Showing segment information for segment #15, Glycine
The segment had been mutated as its original amino acide was Iosleucine

This note has information about the protein at the segment hovered. It has the segment number and the amino acid of the segment. It has the segment score. It also has a list of additional values: clashing,
pairwise (sometimes), packing, hiding, bonding, sidechain, and reference. This information can be obtained using Lua V2.0 objects and functions.

  segmentindex = 15
  current.GetSegmentEnergySubscore(segmentindex, "packing")

will return the packing value of segmentindex. The image above shows a packing score of 34.2.

Segments are referred to by the term segmentindex. This number runs from 1 to the number of segments in the protein.

structure.GetCount()returns the number of segments in the protein.

structure.GetAminoAcid(segmentindex)returns a single lowercase letter representing the amino acid for a particular segment index. The letter follows the standard as shown at this link http://en.wikipedia.org/wiki/Amino_acid and http://foldit.wikia.com/wiki/Lua_Script_Library

If there are ligands in the puzzle, then the ligand is given a secondary structure with the capital letter 'M, and it located at the last segment of the protein. One needs to decrease the segment count by one if a ligand exists.

function filib.GetProteinSegCount()
  segCount = structure.GetCount()
  -- ligand is 'M' at the last segment of the protein
  isLigand =(structure.GetSecondaryStructure(game.segCount))
  if isLigand == 'M' then
    segCount = segCount - 1
  end
  return segCount
end

Fold.it Lua V2 Objects

Fold.it Lua V2 uses objects to group functions. The list of objects are:
The objects and their functions fall into three broad categories,
  • those functions that get information ("getters)
  • those functions that set information ("setters)
  • those functions that physically change the puzzle ("doers").
The objects that strictly retrieve information ("getters" only) are:
Note: The 'puzzle' object has one function that does change the puzzle ("doer"), StartOver().

The objects that get and set information ("getters and setters") and are "doers" are:
  • behavior
  • dialog
  • recipe
  • save
  • structure
  • freeze
  • band
The structure and rotamer objects retrieve information of the protein and
provide the functions to manipulate the protein directly.

The contactmap object is used during exploration puzzles.

   [Don't know if the script can determine if the puzzle is an explorations puzzle?]

The recipe and save objects provide storage of information to measure progress and restore previous poses.
The behavior object is the same as the Behavior/Clash Importance GUI feature.
The freeze and band objects add additional items to the puzzle to help manipulate the protein, similar to the GUI freeze and band features.
The dialog object helps the Lua V2.0 script writer communicate with the user. This is a new feature with version 2.0

GUI Features and Lua functions

The Fold.it functions provide access to most, but not all of the same GUI features found in the client.

The GUI controls that are not available in Lua V2.0 are:
  • The 'Undo' GUI feature is not available in the Lua V2.0. One cannot go back through the 'undo' list using Lua V2.0 scripts.
  • Tweak features are not available with Lua V2.0
As of 9Jul2013, a band cannot connect to an adjacent segment.

It will be noted when a Lua function is the same as a GUI control.

A comparison table of GUI and Lua features would be handy. I should add such a table later.

For the most current, but not detailed documentation of functions,
open the recipe editor, click on New (ScriptV2.0), then click on the
Help icon.

Also the following recipe lists Fold.it Lua functions, with a brief description. http://fold.it/portal/recipe/46122 

Print and Help()

There are two miscellaneous functions, print() and help().

print() outputs to the recipe window.

print() uses the following syntax. Each argument is separated by a comma. The '+' (append) operator is not available as is found in standard Lua.

print(segmentscore," ",creditbest.GetScore())

Anytime print() is called, the string is shown on the recipe output window. It is also written to file. On Windows 7 the file is in C:\foldit\scriptlog.default.xml.

When using tracks, the name of the track will be used for the file. If the name of the track was Wigglefast, then the log file would be C:\foldit\scriptlog.Wigglefast.xml.

help() outputs a list of the API functions. No descriptions.

Shows number output generated by print().
Shows start of function output generated by help()


Puzzle Score

There are four objects that refer to the puzzle's score. They are 'getters' only.
  • absolutebest
  • creditbest
  • current
  • recentbest
Each of these four objects have slightly different list of functions.
The functions below are marked with AB, CR, CU, RE, or ALL. This shows
which object supports that function.

The absolutebest, creditbest, and current objects more or less stay in sync if one is on-line. They will be different if one is working on a puzzle off-line.

[Need to document what happens with these objects when one is off-line]

AreConditionsMet() [ALL] - This is the same as at the top of the client window. Will always return true when 'No bonuses or conditions' are show in the scoring window at the top of the client screen.


This text 'No bonuses or conditions' always returns true.


GetEnergyScore() [ALL] - This is the score shown in the score window at the top of the client.

GetExplorationMultiplier() [ALL] - Some puzzles have an exploration multiplier. This function returns that multiplier. If not an exploration puzzle, it returns 0.
GetScore() [ALL] - This is the same as GetEnergyScore(). Not sure why there are two functions. Technically the energy score gets smaller as the protein is folded more towards its natural shape.

GetSegmentEnergyScore(segmentindex) [ALL] - Get the Rosetta energy score for segmentindex. The sum of all segment scores will be 8000 points lower than the GetScore().

    segscore = 0
    for i=1,structure.GetCount(),1 do
      segscore = segscore + creditbest.GetSegmentEnergyScore()
    -- 8000 difference between scores.
    print(segscore," ", creditbest.GetScore())

GetSegmentEnergySubscore(segmentindex,part) [ALL] - Get the energy score of a part of the segment. The part argument may be one of the following strings.
  • "clashing"
  • "packing"
  • "hiding"
  • "bonding"
  • "backbone"
  • "sidechain"
  • "reference"
  • "disulfides"
  • "density"
This is the same list as seen in the Note when the GUI feature <tab> key is pressed when hovering over a segment. See above.

function filib.reportsegmentparts(segmentindex)
  local parts =
    {"clashing","pairwise","packing","bonding","hiding",
     "sidechain","reference","disulfides","density"}
  report  = "segment#: "..segmentindex.." \n"
  segment_total = 0
  for i=1,#parts,1 do
    partvalue = 
      recentbest.GetSegmentEnergySubscore(segmentindex,parts[i])
    report = report .. "  "..parts[i]..": "..partvalue.." \n"
    segment_total = segment_total + partvalue
  end
  print(report)
end

Restore() [AB,CR,RE] - Load the best score of the appropriate object. Just like the GUI control.
Save() [RE] - Save the current pose as the best for recentbest only. Just like the GUI control.

All of the scoring functions are 'read-only'. The values can only be read, not changed. That would be cheating. :) 


puzzle.GetDescription() - Returns the text description of the puzzle as seen in the comment  section of the Select Puzzle window. The text is one long string.




puzzle.GetExpirationTime() - Can compare to current time to find out how much time is left. This time is GMT. Below is the code to display the time left, as is shown at the top of the client in the score window.

expire = puzzle.GetExpirationTime()
now = os.time()
timeleft = expire-now

print("Time left in seconds ",timeleft)
days = math.floor(timeleft/(60*60*24))
timeleft = timeleft - days*60*60*24
hours = math.floor(timeleft/(60*60))
timeleft = timeleft - hours*60*60
minutes = math.floor(timeleft/60)
timeleft = timeleft - minutes*60
seconds = timeleft 

daystring = string.format("%2d",days)
hourstring = string.format("%2d",hours)
minutestring = string.format("%2d",minutes)
secondsstring = string.format("%2d",seconds)
timestring = "Time left on puzzle: "..daystring..
  " days "..hourstring..":"..minutestring..":"..secondsstring
print(timestring)

puzzle.GetName() - A string is return holding the name of the puzzle.
puzzle.GetPuzzleID() - The ID of the puzzle is returned. Not certain how to use this.
puzzle.StartOver() - This is an action, not a read. It changes the puzzle. It resets the puzzle to the start pose.


scoreboard.GetGroupRank() - Can determine if recent recipe action has raised the ranking, or the ranking has fallen during the time the script was running. One's ranking can fall even if the score increased.
scoreboard.GetGroupScore() - Get group score. This is the best group score for this puzzle.   Not necessarily the absolutebest.GetScore() of the puzzle.
scoreboard.GetRank(scoretype) - Get rank of this puzzle.
  scoretype is a number from 0 to 3.
  • 0 - SCORE_SOLOIST
  • 1 -SCORE_EVOLVER
  • 2 - SCORE_ALL_HANDS
  • 3 - SCORE_NO_SCORE
[Note: as of 04Aug13, 2 and 3 return zero]

scoreboard.GetRank(scoreboard.GetType())
    Solo rank if puzzle is being run solo. Group rank if puzzle is being evolved.
scoreboard.GetScore(scoretype) - Get the score based on type. 
scoreboard.GetScoreType() - Return the scoreboard type.
    Thus scoreboard.GetScore(scoreboard.GetScoreType()) returns
the current score for active puzzle.


The user object holds information about the user.
GetGroupID(), GetGroupName(), GetPlayerID(), GetPlayerName().
Again read only functions. These values are changed at the http://fold.it website, under the user's profile.


The recipe object is helpful to output information to the user as the script runs.
CompareNumbers() - Return true if two real numbers are the same to the precision Lua prints. This is a useful function to determine if there has been any change in the score.
GetRandomSeed() - The value returned from this function can be input into the
    standard lua function math.randomseed().
    WARNING- Please read the following two links before generating random numbers.

To make sure random numbers are generated, add this code to the start of each script that requires random numbers. Seed the random number generator and ignore the first random number.
  math.randomseed( os.time() )
  math.random()

The three functions SectionStart(), SectionEnd(), and ReportStatus() create three XML elements:
<Section name="startname">, </SectionName>, and <SectionStatus ...>, respectively.

ReportStatus() - Reports status to recipe output window and the recipe log file.
    Returns a lot of information.
SectionStart(sectionname) - Marks a section with a string name. 
SectionEnd() - Returns the scoring difference between now and SectionStart(). This does not disturb recentbest, absolutebest, creditbest values.

The output of  SectionStart()SectionEnd(), and ReportStatus() are written to the file script.default.xml. This file is same file that the output of the print() function is recorded in. See print() function above.

save

The save object is helpful to create snapshots of the protein. There are 100 slots allowed.
Quickload(n) - Load snapshot in slot n.
Quicksave(n) - Save snapshot into slot n.
LoadSecondaryStructure() - Restore secondary structure stored by SaveSecondaryStructure().
SaveSecondaryStructure() - This saves the secondary structure. structure.SetSecondaryStructure(...)changes the secondary structure. Only one save slot for secondary structure.

structure

The structure object, the rotamer, and the contactmap objects tells you about the protein.
Now we're getting somewhere. :)
The functions of the structure object that retrieve information about the protein ("getters") are:
CanMutate(segmentindex,aminoacid) - Returns if a specific segment can be mutated to a specific amino acid. The aminoacid value is one of the standard one letter representations for amino acids.See http://en.wikipedia.org/wiki/Amino_acid
As of 14Mar2013,
 Virtual residues will now return 'vrt'
 Sugars will now return 'unk'
 DNA will now return 'da' for "ADE", 'dc' for "CYT", 'dg' for "GUA", and 'dt' for "THY"
 RNA will now return 'ra' for "RAD", 'rc' for "RCY", 'rg' for "RGU", and 'ru' for "URA"

GetAminoAcid(segmentindex) - Returns the aminoacid letter or letters.
GetAtomCount(segmentindex) - Returns the number of side chain atoms in a segment. For instance, glycine has no side chain atoms, so the count is zero.
GetCount() - Get the number of segments in the protein. Almost all scripts use this function. This count is for the current count. The count can change in design puzzles.
  [don't know if the script can determine if the puzzle is a design puzzle]
GetDistance(segmentindex1,segmentindex2)- Get the distance between two  segments. 
GetNote(segmentindex) - Gets the text notes of the segment.
GetSecondaryStructure(segmentindex) - Returns E=sheet, H=helix, L=loop, M=molecule.
    Note: UPPER case letters for secondary structure, lowercase for amino acids.
IsHydrophobic(segmentindex) - Is the segment index hydrophobic or hydrophilic? True- the segment is hydrophobic.
IsLocked(segmentindex) - Is the segment index locked? In some puzzles, part of the segments are locked. This can allow the script to skip segments that can't be changed.
IsMutable(segmentindex)- Returns whether a segment is mutable.

Now for the structure object functions that set information about the protein ("setters")
SetAminoAcid(segmentindex, aminoacid) - Change the segment index to the new aminoacid.
SetAminoAcidSelected(aminoacid) - Change all selected segments to the new aminoacid.
SetNote(segmentindex,string) - Place string into note section of segment.
SetSecondaryStructure(segmentindex,secondarystructure) - Change the segment index to the secondary structure, e=sheet, h=helix, l=loop, m=molecule. Letter can be uppercase or lowercase.
SetSecondaryStructureSelected(secondarystructure) - Change all selected segments to the secondary structure.

Lastly, the structure object functions that change the protein ("doers").
IdealizeSelected() - Idealize the selected segments. The selection object is used to select the segments idealized.
LocalWiggleAll(iterations,[backbone, sidechains]) - Does a local wiggle of all segments for iterations. Can be set to wiggle on the backbone or the side chains for both. Note LocalWiggleAll() is not the same as WiggleAll().
LocalWiggleSelected(iterations,[backbone,sidechains]) - Does a local wiggle of the selected segments. Can be set to wiggle the backbone or the side chains for both.
MutateSidechainsAll(iterations) - The same as the GUI 'Mutate'.
MutateSidechainsSelected(iterations)- Mutate selected segments' side chains for iterations. The selection object is used to select the segments mutated.
RebuildSelected(iterations) - Rebuild selected segments for iterations. The selection object is used to select the segments rebuilt. The GUI Rebuild automatically selects all the segments of the selected secondary structure.
ShakeSidechainsAll(iterations) - Same as GUI Shake.
ShakeSidechainsSelected(iterations)- Shakes the selected side chains.
WiggleAll()- Same as GUI Wiggle All.
WiggleAllSelected()- Wiggle All selected segments. The selection object is used select segments to wiggle.

rotamer

The rotamer object has two functions, one "getter", one "setter".
GetCount(index) - This is the number of rotamers the sidechain at index can snap to.
SetRotamer(index, position)- This sets the rotamer position for the given index. Position can be from 1 to rotamer.GetCount().
behavior

The behavior object modifies how shake and wiggle operate, "getters", and "setters".
GetClashImportance() - This gets the clash importance, or CI, a value from 0 to 1.0.
SetClashImportance(ci) - Set the CI to a value from 0 to 1.0.
GetBandStrengthMultiplier() - Get the band strength multiplier.
SetBandStrengthMultiplier(factor) - Sets the band strength multiplier, 0.5 to 1.5, +/- 50%
GetSlowFiltersDisabled() - Get true if slow filters are disabled. See SetSlowFilterDisabled()
SetSlowFiltersDisabled() - Same a GUI Behavior/Disable Slow Filters. If on, skips CPU intensive filters. Must be enabled to credit score. Disabling slow filters will make AreConditionsMet() return false.

contactmap

The contactmap object is only used for exploration puzzles, ("getter").
IsContact(index1,index2) - Are two segments in contact (within a specified minimum)?
GetHeat(index1,index2) - Get the heat of two segments. This returns a value from 0 to 1.
   1 indicates all users have this pair of segments in contact.
   0 indicates no users have this pair of segments in contact.
   Between 0 and 1 is the percentage of users that have this pair of segments in contact.
   
One would expect GetHeat(n,n+1) to always return 1 for all segments of the protein.
On a non-exploration puzzle IsContact(n,n+1) returns false.

selection 

One of the features of a Lua V2.0 script is to be able to manipulate the protein one segment or more
at a time. In the GUI client, one makes selections with the shift key down for freezing
one segment and right-click to select an entirestructure. Lua V2.0 can be more detailed
than the GUI.

The selection object provides the functions to select, and deselect segments for manipulation
by the structure.xxxSelected() functions.

The segments are selected using these functions.
Select(index) - Select a segment index, just one.
SelectAll() - Select all the segments, 1 to structure.GetCount()
SelectRange(index1,index2) - Select from segment index1 to segment index2, inclusive.
Deselect(index) - Deselect segment index.
DeselectAll() - Deselect all segments.

No. There is no DeselectRange(index1,index2).

The selected segments do not affect the structure.xxxAll() functions.
Only the structure.xxxSelected() functions act on the selected segments.
There is only one set of selected segments.

One has to create storage in your Lua V2.0 script to hold more than one list
of segments to be selected and call DeselectAll() each time you want to use it.

freeze

The freeze object modifies the properties of the protein. It is just like the GUI feature.
A Lua V2.0 program must select a sub-structure to freeze the whole sub-structure.
The GUI just requires a right-click followed by a freeze.
The freeze functions are:
Freeze(index,backbone,sidechain) - The segment index is frozen. The backbone may be frozen. The sidechain may be frozen, or both. If neither are selected, nothing happens.
FreezeAll() - Freezes all segments, both backbone and sidechains.
integer, integer GetCount() - Get the count of the number of segments frozen. This is an unusual function, as it returns two values instead of one. The first value is the backbone frozen count. The second value is the sidechain frozen count.
boolean, boolean IsFrozen(segment index) - Is the backbone or sidechain of the segment index frozen?
UnfreezeAll() - Unfreeze all backbones and sidechains.

band

The band manipulates the bands. In the GUI, if one right-clicks on a band,
one sees the list of options, enable/disable, set length, set strength, delete.
These options are also available in the band  object, and with more range
to strength and length.



Bands are numbered starting at 1. Each time a band is created a number is assigned.
The function that created the band returns the band number of the new band.

The default length of a band is 3.5. The default strength of a band 1.0.
Call SetGoalLength() and SetStrength() to change these values after creating a band.

When a band is created it is automatically enabled. Call Disable() to disable band.

Add(segmentOrigin, segmentXaxis, segmentYaxis, rho, phi, [atomindexorigin],[atomXaxis],[atomYaxis])
    This produces a band that ends in empty space. This is the only 3-D function in Lua V2.
AddBetweenSegments(index1,index2,[atomindex1],[atomindex2])
    A band is drawn between the two segments or between the two atom indices
of the two segments. The band length is structure.GetDistance(index1,index2) or close there to.
    AddBetweenSegments(4,10,1,2) will create a band between segments 4 atom index 1 and segment 10 atom index 2. The maximum value of atom index for segment 4 is structure.GetAtomCount(4) and for segment 10 is  structure.GetAtomCount(10).
AddToBandEndpoint(segmentindex, bandindex,[atomindex])
    A band from segment index or atom index of segment index to the endpoint of bandindex band. This only applies to bands created that end in empty space.
Delete(bandindex) - Delete band index band.
DeleteAll() - Delete all bands. Same as GUI Remove All Bands.
Disable(bandindex) - Disables band index band. Same as right-click on band, disable.
DisableAll() - Disable all bands.
Enable(bandindex) - Enable band index band. Same as right-click on band, enable.
EnableAll() - Enable all the bands.
GetCount() - Get the count of the number of bands.
GetGoalLength() - This is the 'true' length of the band, at which length the band
    neither pushes or pulls. See SetGoalLength().
GetLength(bandindex) - Get the current length of the band.
GetStrength(bandindex) - Get the strength of the band. See SetStrength()
IsEnabled(bandindex) - Is the band enabled?
SetGoalLength(index,length) - Set the 'true' length of the band. A value from 0 to 1000.
    If the band is less than this length, GetLength(index) < GetGoalLength(index), then  the band pushes the endpoints apart. If GetLength(index) > GetGoalLength(index), then the band pulls the endpoints inward.

SetStrength(bandindex,strength) - Set the band strength. 

ui

The ui dialog allows the Lua V2.0 scripts to move the puzzle around on screen in a limited way.
AlignGuide(segmentindex) - This is the same as the GUI feature. Align the protein with
    the guide at the segment index.
CenterViewport() - This is the same as the GUI feature, HOME. Useful to make
    sure the user can see the protein after many iterations.
GetPlatform() - Gets the 'OS' the client (script) is running on.
    win_x86 is one response.
GetTrackName() - Gets the track name. This gets the name of the current track.
    GUI allows user to create independent tracks. Since each track must be unique from puzzle to puzzle, some naming convention must be agreed to with the user so that the script doesn't have to change with each puzzle if the script is to use a particular track.
puzzle.GetPuzzleName() or puzzle.GetPuzzleID() could provide a 'root' and then append standard suffixes for all puzzles. Without a GetTrackNameList() or a SetTrackName(), this function seems limited without a convention in track names.

dialog

Finally, the dialog object. This is the most complicated Fold.it Lua V2.0 object.
The functions are required to be called in a specific order.
CreateDialog() - Call this function first. It returns a lua table that is used
    with the rest of the 'dialog' functions.

One can add to the dialog window sliders, checkboxes, labels, and buttons.

AddLabel(string)
AddCheckbox(string,[true,false]) - If 'true', box is checked by default
AddSlider(string,default,low,high,precision) - A slider is presented that allows the user to select a number from 'low' to 'high' with a decimal precision of 'precision'. Use 0 for precision when working with integers. 'default' is the value shown at the start.
AddButton(string,value) - Each button needs to have its own unique value.
    dialog.AddButton("OK",1)    dialog.AddButton("Cancel",0)
CreateDialog(string) - A dialog object is created that has labels, checkboxes, sliders and buttons added to it. dialog.show(dialog) is used to display the dialog object created by dialog.CreateDialog(string)
Show(dialog) - Here dialog.Show(dialog) receives the variable of dialog.CreateDialog(string). The return code of dialog.Show(dialog) will be the button selected.

The dialog window is a finite width and maximum height. It maybe necessary to program a 'Next' button to provide the user with another  window of selections.

A Glimpse into the Future

With a little investigating, one is able to find that there are several functions that are available to be called, but do not do anything.

One can only speculate what the functions will do. Get and Set give some hint. structure.Tweakxxx() is obvious. behavior.xxxAccuracy() would seem to be a feature to make wiggle and shake run until the change in points is within a range and simply wait however many iterations it takes to get there.

The following code will expose the functions of each of the major objects.

for key,value in pairs(object) do
    print("found member " .. key);
end

where object is one of the Fold.it Lua 2.0 objects, such as structure, band, or current.

These functions are:

structure.TweakRotate(segmentindex,angle) - Rotate segment by angle, angle in radians, -pi to +pi. Segment must be a sheet or helix.
structure.TweakShift(segmentindex,shiftDirection) - Shift segment index one way or other. Segment must be a sheet.
structure.TweakStraighten(segmentindex) - Straighten segment index. Segment must be a sheet.
behavior.GetBuriedSideChainAccuracy() - ??
behavior.SetBuriedSideChainAccuracy() - range 0 to 1
behavior.GetExposedSideChainAccuracy() - ??
behavior.SetExposedSideChainAccuracy() - range 0 to 1
behavior.GetShakeAccuracy() - Returns the shake accuracy
behavior.GetWiggleAccuracy() - Returns the wiggle accuracy
behavior.SetShakeAccuracy() - Sets the shake accuracy. range ?
behavior.SetWiggleAccuracy() - Sets the wiggle accuracy. range ?


Saturday, August 3, 2013

Augment GDB with Convenience Variables and User-defined Commands

I have had the need over the past few months to work with GDB but without a symbol table.

Searching the web one finds others, though a small minority, that have asked how to work with GDB when there is no symbol table available.

I posted several questions to Stack Overflow regarding GDB and not having a symbol table.
SO Question 17956799
SO Question 17492119
SO Question 17956799
SO Question 17667590

People have reported problems of not being able to set breakpoints. GDB reports
     'No symbol table loaded. Use "file" command'

SO Question 4698299

Some of these questions are from newbie programmers who didn't include debug symbols in their build, but others are those, like me, trying to work with legacy programs that are no longer compatible with the GDB tool chain.

This is typical in reverse engineering of legacy code.

This blog tells about how I made GDB useful when there is no symbol table available. My experience is in the embedded environment. I have small memory sizes, slow (relatively) speed processors, and only one process. I am also working with cross-tools and target hardware.

Having used GDB before will help when reading what follows. I have added links to GDB commands mentioned in the text.

Symbol Map

First, one needs to have a symbol map.

If one doesn't have a symbol map, well, you can start creating your own by giving your own names to hex addresses where call or jsr instructions are found in a dis-assembly listing, but having a symbol map is much better.

What information does a symbol map have? Addresses with associated names.

It is much better to refer to a memory address as main, than 0x243AD. (Though over time you might start seeing main, when you see 0x243AD, but why bother?)

A symbol map, however, is not a symbol table, as far as GDB is concerned.

I have not found any documentation on how to create a symbol table file without using a linker.

One is able to create convenience variables in GDB that hold the addresses in the symbol map and give these addresses names that GDB can use. GDB Convenience Variables

From my symbol map, I have this record.

_CheckCheckSum__6EepromFs        code      000220D2    eeprom
Eeprom::CheckCheckSum(short)

I can create a GDB convenience variable.

set $Eeprom_CheckCheckSum=(unsigned int*)0x220D2

I can then say

x /40i $Eeprom_CheckCheckSum

The GDB examine (x for short) command will disassemble the next 40 instructions starting at the address $Eeprom_CheckCheckSum.
GDB examine command

JTAG or BDM Device

If one has legacy code, it probably doesn't support GDB. To use GDB, one MUST have a GDB server, either in the target hardware or on a device connected to the target hardware.

Current JTAG or BDM (Motorola CPUs only) devices provide the GDB server.

When using a JTAG device, the target of GDB is considered remote. GDB has a configuration command, target remoteGDB target remote command

Once GDB can communicate with the JTAG device, then GDB has a command that allows one to send JTAG device-specific commands, monitor.

Any command that you want to send to the JTAG device is sent via GDB by

 monitor <JTAG command>

mon is short for monitor.

Example, to reset the CPU via the Abatron BDI2000, one issues the GDB command

mon reset

Abatron BDI200 User's Manual

GDB command file

When GDB is started, it looks for the file .gdbinit. This file name can be overridden using the -x <filename> command line option. GDB Startup

This is the file where we place the target command to communicate with the JTAG device or GDB server. Placing the target remote command in .gdbinit ensures the GDB session is communicating with the target. If there's a communication problem, we will know it right away.

# Communicate with an Abatron BDI2000
target remote 192.168.0.10:2001

.gdbinit is also where we place the convenience variables we derived from the symbol map.

From my symbol map, I have this record.

_CheckCheckSum__6EepromFs        code      000220D2    eeprom
Eeprom::CheckCheckSum(short)

I create the following convenience variable and put it in the .gdbinit file.

set $Eeprom_CheckCheckSum=(unsigned int *)0x220D2


One can list the convenience variables with the GDB command

show conv


GDB User-defined Command

Along with convenience variables, GDB also allows one to define user-defined commands.

User-defined commands are way to create automated functions for repetitive tasks or complicated debugging logic.

They can also be used to create short cuts for JTAG device commands.

Below are a few of Abatron BDI2000 JTAG commands and the GDB equivalent.

# BDI2000 command 'reset'
define mreset
 mon reset
 flushregs
end
document mreset
Send the BDI2000 command 'reset'
'flushregs' is required as GDB does not
know what 'mon reset' did to the target
end

# BDI2000 command 'mrd'-read registers
define mrd
mon rd
end
document mrd
Send the BDI2000 command 'rd'. More compact than 'info reg'
end

In the document section for the mreset user-defined command is a key concept. If you use ANY monitor command that changes the registers of the target CPU, you MUST include the GDB flushregs command at the end of the command.

GDB does not know what monitor commands do to the CPU state, flushregs is the GDB command to say to GDB 'The state of the registers has changed, your cache is no longer valid.'

The user-defined commands are also placed in .gdbinit.

show user [command name]

The show user command lists the user-defined commands.
GDB show user command

So with the above user-functions and environment variables, one change enter at the GDB command prompt.

mreset
mrd
x /40i $Eeprom_CheckCheckSum

If the target hardware has hardware breakpoint support, I can also say

br $Eeprom_CheckCheckSum
continue

When needing to disassemble instructions constantly, I created a user-command dxi.

# dxi
# dxi <count>
# dxi <count> <startaddress>
define dxi
if $argc == 2
  x /$arg0i $arg1
end
if $argc == 1
  x /$arg0i $pc
end
if $argc == 0
x /20i $pc
end
end
document dni
Output $arg0 instructions from current PC
If $arg0 is not supplied, output 20 instructions
end

If one wants to capture the dis-assembly of the examine command, one can create a pair of user-commands to turn on and turn off logging. The dis-assembly listing will be recorded in a log file.

#startlog
#startlog <file>
define startlog
  if $argc == 0
    set logging off
set logging on
  end
  if $argc == 1
    set logging off
    set logging file $arg0
    set logging on
  end
end
document startlog
  Open 'gdb.txt' or $arg0 for GDB logging
end

define stoplog
set logging off
end
document stoplog
Turn off GDB logging
end

Thus without a symbol table one can create a usable debugging environment.

Hope you found this post worth reading.

The next post discusses how to augment GDB when the target processor does not have hardware breakpoint support.