пятница, 22 июля 2016 г.

ObjectTravel. A tool to traverse object references

ObjectTravel is a tool to deeply traverse "native" references of objects through instance variables and "array contents".

     Gofer it
          smalltalkhubUser: 'Pharo' project: 'ObjectTravel';
          configuration;
          loadStable

Usage is quite simple: just create traveler instance on your object and send #referencesDo:

      traveler := ObjectTraveler on: (1@2 corner: 3@4).
      traveler referencesDo: [:eachRef | eachRef logCr].

It will print on transcript:

      (1@2)
      (3@4)
      1
      2
      3
      4

There is convenience method to collect all references:

      traveler collectReferences. "==> an OrderedCollection((1@2) (3@4) 1 2 3 4))

Traveler enumerates each reference in breadth-first direction and visit every reference only once:

      point := 1@2.
      traveler := ObjectTraveler on: {point . #middle. point}.
      traveler collectReferences "==> an OrderedCollection((1@2) #middle (1@2) 1 2)"

"point" is referenced two times. But traveler will go deeply only once. So circles are supported:

      array := Array new: 2.
      array at: 1 put: 1@3.
      array at: 2 put: array.
      traveler := ObjectTraveler on: array.
      traveler collectReferences asArray = { 1@3. array. 1. 3 } "==> true"

You could control traveler to skip particular objects:

      rect := 1@2 corner: 3@4.
      traveler := ObjectTraveler on: rect.
      traveler skip: rect origin.
      traveler collectReferences "==> an OrderedCollection((1@2) (3@4) 3 4)"

(it can be done during enumeration too).

Also you could apply predicate for objects which you want traverse:

      rect :=
      traveler := ObjectTraveler on: {1@2 corner: 3@4. 5@6}. 
      traveler traverseOnly: [:each | each isKindOf: Point]. 
      traveler collectReferences "==> an OrderedCollection((1@2) corner: (3@4) (5@6) 5 6)"

There are another convenient methods:

      traveler countReferences

It will count all references in object graph.

      traveler copyObject

It will make deep copy of original object which is similar to #veryDeepCopy: cyclic references are supported. 

Also traveler could find all paths to given object in object graph:

      root :=  {#one -> #target. {#emptyArray}. {#two -> #target}}.
      traveler := ObjectTraveler on: root.
      traveler findAllPathsTo: #target "==>
           {#one -> #target. root}
            {#two -> #target. {#two -> #target}. root} "

During enumeration traveler could replace visited references:

      rect := 1@2 corner: 3@4.
      traveler := ObjectTraveler on: rect.
      traveler referencesDo: [:each | 
           each isInteger ifTrue: [traveler replaceCurrentReferenceWith: each + 1]].
      rect "==> 2@3 corner: 4@5"

It would be fun project to connect ObjectTraveler to XStreams to make any object "streamable":

      (1@2 corner: 3@4) traveling reading count: [:each | each isInteger].
      (1@2 corner: 3@4) traveling reading get; get; read: 2; rest.
      (1@2 corner: 3@4) traveling writing write: 100@500

четверг, 21 июля 2016 г.

Major Seamless update

I glad to publish new version of Seamless (0.8.2) together with new doc.
It could be loaded by:

       Gofer it
       smalltalkhubUser: 'Pharo' project: 'Seamless';
       configuration;
       loadStable

It works in Pharo 5 and 6. As usually feedback is welcome.

It is complete redesign of original version with the goal to make it more flexible, reliable and simple. (original version was created by Nikolaos Papoulias).

New version introduces new project Basys as foundation for new design.
Basys implements an abstract layer for networks which require bidirectional communication between clients and servers. (details here).
Seamless implements Basys network to organize transparent communication between distributed objects.

To use Seamless SeamlessNetwork should be created on client and server:

       network := SeamlessNetwork new.

To accept connections server should be started:

       network startServerOn: 40422.

Clients could connect to server and retrieve remote environment:

       remotePeer := network remotePeerAt: (TCPAddress ip: #[127 0 0 1] port: 40422).
       remoteSmalltalk := remotePeer remoteEnvironment.

       "or short version"
       remoteSmalltalk := network environmentAt: (TCPAddress localAt: 40422)

remoteSmalltalk here is proxy which delegates any received message to remote object. Remote messages are executed on server which return result back to client. Result can be returned as another proxy or as copy which contains another proxies.

In example result is reference to remote Smalltalk instance. It can access globals from remote environment:

       remoteTranscript := remoteSmalltalk at: #Transcript.
       remoteTranscript open; show: 'remote message'; cr

It will open transcript on server and print text on it.

Arguments of remote message are transferred to server with same logic as message result transferred to client. On server arguments can include proxies and server can send messages to them:

       remoteTranscript print: #(1 2 3 4 5)

Here array will be passed to server as reference. Then on server transcript will interact with it to print it. And as result client will receive messages from server.

Concrete transfer strategy is depends on transferred object. It is specified in method #seamlessDefaultTransferStrategy:

       Object>>seamlessDefaultTransferStrategy
            ^SeamlessTransferStrategy defaultByReference

       Number>>seamlessDefaultTransferStrategy
            ^SeamlessTransferStrategy defaultByValue

Default strategy could be overridden on network level:

       network transferByValue: (Instance of: Point).
       network transferByReference: (Identical to: #specificSymbol)

It allows to tune network for specific application to optimize communication between distributed objects. There are few other transfer strategies which allow minimize remote communication. Most interesting allows properties caching. It will transfer object reference together with specified properties. Following example will configure network to transfer class reference together with name:

       network transferByReference: (Kind to: Class) withCacheFor: #(name)

And then proxy for remote class will return #name from local cache instead of real remote call.

Previously announced project RemoteDebuggingTools uses all these feature to minimize communication between debugger and remote process. It allows to reduce number of remote messages for opening debugger from 800 to 13.

Now another important feature: block evaluation on remote peers:

       remotePeer evaluate: [1 + 2]. "==>3"

Given block is transferred to remote side and evaluated. Result is returned to client. As in other cases it could be proxy or normal object.

Block could use globals. On remote side they will be local globals of this remote environment. Following example will show notification on remote image:

       remotePeer evaluate: [Object inform: 'message from remote image'].

Temps and workspace variables can be used too. They will be transferred according to own strategies:

       | result |
       result := OrderedCollection new.
       remotePeer evaluate: [100 to: 500 do: [:i | result add: i factorial ]].

Non local return is also supported in regular Smalltalk semantics:

       remotePeer evaluate: [1 to: 10 do: [:i | i>5 ifTrue: [^i] ] ]. "==>6"

Also block could be evaluated asynchronously without waiting any result:

       | result |
       result := OrderedCollection new.
       remotePeer evaluateAsync: [result add: 1000 factorial]. "it will not wait result"

Seamless provides integration with GT tools. Remote proxies could be inspected to explore remote objects state with ability to execute remote scripts (doIt, printIt). It is shown on remote debugging demo.

To analyse remote communication Seamless implements special tool SeamlessLogger. It is explained in doc.  Here is little example for following code:

       remoteRect := remotePeer evaluate: [0@0 corner: 2@3].
       remoteRect area. "==>6".
       localRect := 0@0 corner: 5@2.
       remoteRect evaluate: [remoteRect area + localRect area] "==>16"

Statistics will show number of messages, receivers and bytes which was transferred over network in dimension of receiver class or message selector:


At the end I must say about two important issues which will be covered in next versions:

1) No garbage collection

SeamlessNetwork keeps all objects which was transferred by reference. They will never be cleaned while network is live. 
Now It could be done manually by evaluating "network destroy". It will clean all object caches and close all connections. It could be not safe because remote peers could use this objects. Seamless tries to handle it properly with clear errors in such cases. 
In future unused distributed objects will be cleaned automatically.

2) No security: no authorization and encryption. Now for security purpose you need external tools like VPN

понедельник, 18 июля 2016 г.

Remote debugging tools is ready

I glad to release first version of RemoteDebuggingTools project. It allows explore and debug remote images.
Here is demo with debugging Seaside application:


On server side you need to install Server group of project:

Metacello new
        smalltalkhubUser: 'Pharo' project: 'RemoteDebuggingTools';
        configuration: 'RemoteDebuggingTools';
        version: #stable;
        load: 'Server'.

Then server should be started on port where client image could connect:

RemoteUIManager registerOnPort: 40423.

Also image could be started with running server from command line:

./pharo PharoWithServer.image debuggerServer --port=40423

On client side Client group should be installed

Metacello new
        smalltalkhubUser: 'Pharo' project: 'RemoteDebuggingTools';
        configuration: 'RemoteDebuggingTools';
        version: #stable;
        load: 'Client'.

Then RemoteDebugger could connect to target image:

debugger := RemoteDebugger connectTo: (TCPAddress ip: #[127 0 0 1] port: 40423).

It registers a RemoteDebugger as debugger tool on remote image. Any error there will open debugger on client image with remote process stack.

With connected debugger instance you can execute scripts:

debugger evaluateAsync: [ [1/0] fork ].
debugger evaluate: [ 1 + 2 ] "==> 3".
debugger evaluate: [ 0@0 corner: 2@3 ] "==> aSeamlessProxy on remote rectangle".

Async evaluation not returns any result of block. It just transfer block to remote side and not waits any result.
Normal evaluation will wait until block competes on remote side. And result will be returned. There is nice integration with GTTools to inspect remote objects.

Blocks could reference local objects like temps or workspace variables. You can write scripts like:

result := OrderedCollection new.
debugger evaluateAsync: [100 to: 500 do: [ :i | result add: i factorial] ].

Non local return is also supported. Following expression works same way as local block evaluation:

debugger evaluate: [ 100 to: 500 do: [ :i | i > 200 ifTrue: [^i] ]].

This project (with scripting facilities) are based on new version of Seamless which will be announced soon next time. It misses very important feature now: distributed garbage collection. When you finish your debugging session you need manually disconnect your debugger. It will clean all garbage:

debugger disconnect.

Other restrictions which will be covered in future:
  • debugging expressions from debugger is not working (debugIt from context menu)
  • code browse and navigation are not working. Nautilus and MessagesBrowser should be able to work with remote environment.
  • remote stack items are printed with prefix SeamlessProxy. It should be improved
  • printIt inside debugger/inspector should return remote printString instead of aSeamlessProxy.  
  • no security: no authorization and encryption. Now for security purpose you need external tools like VPN

воскресенье, 17 июля 2016 г.

Magic with Pharo Reflectivity

Last days I play with idea of compile time evaluation without special language support.
For example in Dolphin Smalltalk there is special syntax for this:

##(100 factorial)

Factorial will be evaluated at method compilation time and result will be saved as literal. And this precomputed value will be used during method execution.

In Pharo we can do this without special syntax. Thank's to powerful reflection system which called Reflectivity. It allows reflection on the level of AST nodes with different kind of transformations.
With Reflectivity we could substitute particular expression with result when method will be executed first time. It is a bit different than compile time evaluation. But after first method execution it will be same: precomputed value of expression will be stored and used as regular literal.

And now magic:

100 factorial asMethodConst

I was quite impressed when I realize that this approach could work. First I tried to use blocks but it was completely not needed:

Object>>asMethodConst
| constNode link |
constNode := thisContext sender sourceNodeExecuted.
link := MetaLink new
metaObject: self;
control: #instead.
constNode link: link.
^const

#asMethodConst will be executed on result of expression. Trick here is that constNode is "message send" ast-node where receiver is target expression. Metalink with control #instead will substitute it as whole and receiver will not be executed anymore.

Wondering how little code is required. It will be available in Pharo soon (18768).

понедельник, 6 июня 2016 г.

ObjectStatistics. Simple objects analysis tool

I am glad to announce ObjectStatistics tool to analyse set of objects by computing different kind of metrics and look at them from different angles. It implements simplistic OLAP Cube approach for data analysis but in objects space.
Imaging that we have collection of message sends and we want to know number of message sends in dimension of receiver, receiver class and message selector. We have different angles to look at this data: from receiver class to selector and receiver or from selector to receiver class and receiver or any other combination. 
We also could analyze different kind of metrics which could be computed on given objects. It could be number of unique receivers, execution time, executed lines of code, etc.
This package implements computation of object statistics over declared metrics and dimensions space.
For example code profilers could be easily implemented by this tool. Following code will produce statistics for printing SmallInteger 3:
ObjectStatistics>>exampleFlatProfiler
 | stat prev |
  stat := ObjectStatistics new.
  stat countAllAs: 'sends'.
  stat countDifferent: [:context | context receiver ] as: 'receivers'. 

  stat 
    dimension: [ :context | context receiver class ] 
    named: 'classes'; with: [ 
        stat dimension: [ :context | context selector ] 
            named: 'msgs' ];
    dimension: [ :context | context selector ] 
    named: 'msgs'; with: [ 
        stat dimension: [ :context | context receiver class ] 
            named: 'classes' ];
    dimension: [ :context | context receiver class -> context selector ] 
    named: 'leaves'; with: [ 
        stat dimension: [ :context | context sender method ] 
            named: 'sender' ].
 
  prev := nil.
  thisContext sender 
    runSimulated: [3 printString] 
    contextAtEachStep: [:current |
 current ~~ prev & (current sender == prev ) ifTrue: [
  stat accumulate: current].
 prev := current].
   
  ^stat inspect
ObjectStatistics provides GTTools integration which shows computed objects in suitable way:

Usually profilers show execution tree to analyse how many message sends was happen per method call or how many time was spent on each method call. To achieve this ObjectStatistics implements special recursive dimension which produce metrics per children::
ObjectStatistics>>exampleTreeProfiler
| stat prev |
stat := ObjectStatistics new.
stat countAllAs: 'sends'.
stat countDifferent: [:context | context receiver ] as: 'receivers'. 
  
stat 
    dimension: [ :context | context receiver class -> context selector] 
    named: 'calls' recursionUpTo: [:context | context sender].
 
prev := nil.
thisContext sender 
    runSimulated: [3 printString] 
    contextAtEachStep: [:current |
 current ~~ prev & (current sender == prev ) ifTrue: [
  stat accumulate: current].
 prev := current].
   
^stat inspect
And inspector will show regular profiler tree:

(here we also see stack frames which belongs to tool which ran example). 
At the end lets look at methods of some package in multidimensional space:
ObjectStatistics>>exampleMethods
| stat |
  stat := ObjectStatistics new.
  stat countAllAs: 'methods'.
  stat countFunction: [:m | m linesOfCode ] as: 'lines of code'. 
 
  stat 
    dimension: [ :m | m methodClass package name ] 
    named: 'pckgs'; with: [ 
 stat dimension: [ :m | m methodClass ] 
     named: 'classes'; with: [
  stat dimension: [ :m | m selector] 
         named: 'methods']];
    dimension: [ :m | 
 m methodClass package mcWorkingCopy versionInfo author ] 
    named: 'authors'; with: [ 
 stat dimension: [ :m | m methodClass ] 
     named: 'pcks'; with: [
  stat dimension: [ :m | m methodClass ] 
      named: 'classes'; with: [
   stat dimension: [ :m | m selector] 
       named: 'methods']]].

  stat accumulateAll: ((RPackage allInstances 
 select: [ :each | each name beginsWith: 'Athens']) gather: #methods).
  ^stat inspect


You can load code by:
Gofer it 
    smalltalkhubUser: 'Pharo' project: 'ObjectStatistics';
    configuration;
    loadStable

пятница, 15 апреля 2016 г.

Halt next object call

ObjectVirus allows very simple implementation of object centric debugging. We can infect given object by virus which will halt any message send.
Let's create new GhostBehaviour for this:

GHGhostBehaviour subclass: #GHObjectCallHalt
      instanceVariableNames: ''
      classVariableNames: ''
      package: 'Ghost-ObjectCallHalt'

GHGhostBehaviour>>send: aMessage to: anObject
       | method breakpoint |
       anObject recoverFromVirus.
       self halt.
       ^aMessage sendTo: anObject

First we disable interception behaviour to not halt any following messages (to get halt once logic). And then we halt message processing. It will open debugger on any message send. And to access target method "step into" button should be pressed few times in the debugger.
To activate such behaviour on objects we can create suitable method:

Object>>haltOnNextCall
       | virus |
       virus := GHObjectVirus behaviour: GHObjectCallHalt new.
       virus infect: self

And now we can play with this:

      point := 2@3.
      point haltOnNextCall.
      0@0 distanceTo: point

It will open debugger:


Of course we don't want to step into target method manually. We want to see target method just in the debugger. To achieve this we can set up breakpoint on target method and run it instead of halt message processing:

GHGhostBehaviour >>send: aMessage to: anObject
       | method breakpoint |
       anObject recoverFromVirus.
       method := anObject class lookupSelector: aMessage selector.

       breakpoint := Breakpoint new
             node: method ast;
             once.
       breakpoint link condition: [ :receiver | receiver == anObject ] arguments: #(#object).
       breakpoint install.

       ^aMessage sendTo: anObject

And with this change we will see what we want:


Debugger shows Point>>x method because #distanceTo: asks #x and #y of given point to compute distance:


Last thing which is not nice is that we see all message interception logic on the debugger stack view. Fortunately Pharo stack model allows context manipulation and we can really remove unneeded stack elements from execution context:

GHGhostBehaviour >>send: aMessage to: anObject
       | method breakpoint |
       anObject recoverFromVirus.
       method := anObject class lookupSelector: aMessage selector.

       breakpoint := Breakpoint new
             node: method ast;
             once.
       breakpoint link condition: [ :receiver | receiver == anObject ] arguments: #(#object).
       breakpoint install.

       self activateTargetMethodOf: aMessage for: anObject

GHGhostBehaviour >>activateTargetMethodOf: aMessage for: anObject
       | targetContext sender objectClass |
       sender := thisContext.
       [ sender selector == #cannotInterpret: ] whileFalse: [ sender := sender sender ].
       thisContext terminateTo: sender sender.
       "We should not use direct message to object because we propaply already install breakpoint to it"
       objectClass := GHMetaMessages extractClassOf: anObject.
       targetContext := thisContext sender
             activateMethod: (objectClass lookupSelector: aMessage selector)
             withArgs: aMessage arguments
             receiver: anObject
             class: objectClass.

       targetContext jump

It covers our tracks of "halt-virus" infection:


At the end we can extend Pharo debugger with new button "Run to next self send" which will resume process execution and halt on next call to current receiver:

ResumeDebugAction subclass: #GHRunToNextSelfCallDebugAction
       instanceVariableNames: ''
       classVariableNames: ''
       package: 'Ghost-ObjectCallHalt'

GHRunToNextSelfCallDebugAction >>defaultLabel

       ^ 'Run to next self call'

GHRunToNextSelfCallDebugAction >>id

       ^ #runToNextObjectCall

GHRunToNextSelfCallDebugAction class >>gtStackDebuggingActionFor: aDebugger
       <gtStackDebuggingAction>

       ^ (self forDebugger: aDebugger)
             icon: GLMUIThemeExtraIcons glamorousPlay

GHRunToNextSelfCallDebugAction >>executeAction

       self currentContext receiver haltOnNextCall.

       ^super executeAction.

And with these debugger will be looked like:


And if you press new button debugger will be opened on #y method of current point:


Code is hosted in Ghost repository. You can use it in your development process:

Metacello new
       smalltalkhubUser: 'Pharo' project: 'Ghost';
       configuration: 'Ghost';
       version: #development;
       load: 'ObjectCallHalt'