четверг, 16 февраля 2017 г.

Calypso navigation model

This time I am going describe Calypso navigation model.

In Calypso users query environment for specific set of objects.
First you need environment instance. There is global one for current image:
env := ClyNavigationEnvironment currentImage
It is navigation environment which is created over some system environment. In this case it is current image:
ClySystemEnvironment currentImage
System environment models the image. It includes package manager, globals and system announcer. And navigation environment provides interface to query information from it. It organizes cache of all queries.  If you will browse senders of message #do: same result will be returned for second call. Cache optimizes performance and memory usage during navigation.
You can use your own navigation instance instead of global one:
env := ClyNavigationEnvironment over: ClySystemEnvironment currentImage
External libraries can provide own system environment and Calypso will be able browse it. For example new version of Ring allows browse code repository with all Calypso features.

For next steps you need a scope where you will look at environment objects.
It can be scope of full system:
systemScope := env systemScope
Or it can be scope of concrete objects:
packageScope := env selectScope: ClyPackageScope of: {Point package}. 
classScope := env selectScope: ClyClassScope of: {Point. Collection}
With scope you can evaluate queries:
packageScope query: ClySortedClasses 
classScope query: ClySortedMethods
packageScope query: (ClyMessageSenders of: #(do: x))
classScope query: (ClyMessageImplementors of: #(select: y:)) 
Any query returns instance of cursor which provides stream access to result (details below).
Result is represented by requested ClyEnvironmentContent subclass. In first example it is instance of ClySortedClasses which is sent as argument. Query method accepts query instance (subclasses of ClyEnvironmentQuery) or compatible object which implements #asEnvironmentQuery message. And class of result itself plays role of most trivial query ClyEnvironmentSimpleQuery. Simple query is responsible to return all objects accessible from given scope in requested form. For example you can query all classes in hierarchical form:
packageScope query: ClyHierarchicallySortedClasses
Any query is created with requested content or defines default one.  In example senders and implementors use sorted methods by default. But they can use different:
classScope query: (ClyMessageSenders of: #(do:) as: ClyHierarchicallySortedMethods)
As was mentioned above actual result of #query: is cursor object, the instance of ClyEnvironmentCursor:
cursor := classScope query: ClySortedMethods.
Cursor provides stream access to requested content items:
cursor currentItem.
cursor nextItem.
cursor moveTo: itemPosition. 
cursor retrieveAll.
Returned items are not raw objects like methods or classes. Instead they are instances of ClyEnvironmentItem which wraps actual object. Items extend objects with arbitrary set of properties. For example if class has abstract method it can be marked with "abstract property". If method is overridden in subclasses it can be marked by "overridden property". Then tools use these properties to provide specific representation for items. For example browser can show abstract classes with italic font and it can show special icon nearly overridden method.

Computation of properties is slow. Imaging that you look at all classes in system and for each class you want abstract property. It will require scanning almost all methods in system.
Calypso solves this problem in two ways:
  • All queries are cached. Computation will be performed only once
  • Properties are computed lazily when objects are really used.  For example they computed for objects which are shown to user, for only visible part of them.
Lazy computation is hidden from users by cursor instance. Cursor asks observed environment content to resolve small part of items according to current position. Then cursor uses resolved part as items cache. And new portion of items are resolved by demand.

This logic provides important optimization for remote scenario where observed content is remote object. In this case only used part of items is transferred over network. And when next part is needed it is loaded to client. It makes remote browser very fast because only visible part of packages, classes and methods are sent over network and all required information is available in few requests.

Now let's play with example cursor. It can be opened in table morph:
cursor := packageScope query: ClySortedClasses.
dataSource := ClyCollapsedDataSource on: cursor.
table := FTTableMorph new.
extent: 200 @ 400;
dataSource: dataSource;

Cursor provides query API to evaluate new queries with modified parameters.
For example you can ask different kind of result for original query:
cursor := cursor query: ClyHierarchicallySortedClasses

Or you can evaluate new query with extra package:
cursor := cursor queryContentInScopeWith: #'AST-Core' asPackage.

Or you can evaluate new query without existing package:
cursor := cursor queryContentInScopeWithout: #Kernel asPackage.

With cursor you can navigate to the children scope of particular items. For example you can query methods of selected classes:
cursor := cursor query: ClySortedMethods from: {Point. Array}.

You can also evaluate original query in different scope. It allows fetch class side methods of selected classes:
cursor := cursor queryContentInNewScope: ClyClassSideScope. 

All these queries never modify cursor state. They always return new cursor instance which points to new result.

On top of this navigation model Calypso implements tree structure. For example you can look at classes and methods in same table morph:
cursor := packageScope query: ClySortedClasses.
dataSource := ClyCollapsedDataSource on: cursor.
dataSource childrenStructure: { ClySortedMethods }.
table := FTTableMorph new.
extent: 200 @ 400;
dataSource: dataSource;

Tree structure can include many levels. For example packages -> class groups -> classes:
cursor := env systemScope query: ClySortedPackages.
dataSource := ClyCollapsedDataSource on: cursor.
dataSource childrenStructure: { ClySortedClassGroups. ClyHierarchicallySortedClasses}.
table := FTTableMorph new.
extent: 200 @ 400;
dataSource: dataSource;

Also it is possible to show hierarchical items with collapsing buttons:
cursor := packageScope query: ClyHierarchicallySortedClasses.
dataSource := ClyExpandedDataSource on: cursor.
table := FTTableMorph new.
extent: 200 @ 400;
dataSource: dataSource;

At the end let's implement new environment content RandomOrderedMethods:
ClyMethodsContent subclass: #ClyRandomOrderedMethods
instanceVariableNames: ''
classVariableNames: ''
package: 'Calypso-Example'
Each kind of environment content implements set of the building methods depending on what scope it supports. For example ClySortedMethods supports extracting methods from classes and method groups. ClySortedClasses supports extracting classes from packages and class groups.
When simple query is evaluated by given scope it asks requested content to be built from concrete objects. For package scope it will ask content for #buildFromPackages:. For class scope it will ask content for #buildFromClasses:.
For simplicity ClyRandomOrderedMethods will support only class scope:
buildFromClasses: classes
items := OrderedCollection new.
classes do: [ :eachClass |
eachClass localMethods do: [ :each |
items add: (ClyEnvironmentItem named: each selector with: each)] ].
items shuffle
Now look at methods in new form:
cursor := env systemScope query: ClySortedPackages.
dataSource := ClyCollapsedDataSource on: cursor.
dataSource childrenStructure: { ClySortedClasses. RandomOrderedMethods}.
table := FTTableMorph new.
extent: 200 @ 400;
dataSource: dataSource;

"Building" methods are required to be able evaluate simple queries with new implemented form. More advanced queries could require another code. Generally queries implement one method:
  • fetchContent: anEnvironmentContent from: anEnvironmentScope
where they dispatch processing to scope and environment content depending on their logic. For example to be able to use new random order for senders or implementors the method #buildFromMethods: should be implemented by content:
buildFromMethods: methods
items := methods shuffled collect: [ :each |
ClyEnvironmentItem named: each selector with: each]
If you open senders browser:
browser := ClyMethodBrowser browseSendersOf: #do:.

you can switch it to new order:
browser switchToResultContent: ClyRandomOrderedMethods

For more details look at code and read class comments.
Next time I will show how extend browser with new functions.

вторник, 24 января 2017 г.

PharmIDE: Pharo remote IDE to develop farm of Pharo images remotely

I am glad announce first version of PharmIDE project which is complete toolset for remote development of Pharo images. It includes:
  • remote debugger
  • remote inspector
  • remote playground
  • remote browser
Old project RemoteDebuggingTools is deprecated. But it could be used for Pharo 5 images. PharmIDE targets Pharo 6 or later and provides everything which was done in original project.

Server part of project should be installed on target image:
Metacello new
        smalltalkhubUser: 'Pharo' project: 'PharmIDE';
        configuration: 'PharmIDE';
        version: #stable;
        load: 'Server'
Then server should be started on port where client image can connect:
PrmRemoteUIManager registerOnPort: 40423
Image can be saved with running server. It will be automatically started when image restarts. Or you can use command line option for this:
./pharo PharoServer.image remotePharo --startServerOnPort=40423
On IDE image client side of project should be installed:
Metacello new
        smalltalkhubUser: 'Pharo' project: 'PharmIDE';
        configuration: 'PharmIDE';
        version: #stable;
        load: 'Client'
And then you can connect Pharo to remote image:
remotePharo := PrmRemoteIDE connectTo: (TCPAddress ip: #[127 0 0 1] port: 40423)
It registers local debugger and browser on remote image:

  • Any error on remote image will open debugger on client image with remote process stack
  • Any browser request on remote image will open browser on client image with remote packages and classes
  • User requests from server are redirected to client. Any confirm or inform requests from remote image will be shown on client. For example author name request will be shown on client image where user can type own name remotely.

With remotePharo instance you can evaluate scripts:
remotePharo evaluateAsync: [ [1/0] fork ].
remotePharo evaluate: [ 1 + 2 ] "==> 3".
remotePharo evaluate: [ 0@0 corner: 2@3 ] "==> aSeamlessProxy on remote rectangle".
Look at original project post for more details about scripting and inspecting. In this version inspector is improved:
  • PrintIt command shows correct #printString of remote object instead of 'aSeamlessProxy~'. 
  • "Workspace" variables are supported in inspector. You can save temp results in it.
  • Implementors/senders shortcuts will open local browser on remote environment. Any subsequent navigation will be bound to remote image.
Remote playground and remote browser support same set of features. All text editors support same commands like in local environment.

To open browser or playground use:
remotePharo openPlayground.
remotePharo openBrowser
Now #debugIt command and refactorings are not working remotely but they will be supported in future.

Now look at PharmIDE in action:

четверг, 12 января 2017 г.

Calypso update: MethodBrowser and better UI

I glad release new version of Calypso. It includes method browser which makes Calypso complete toolset for system navigation.

Also new version contains many UI changes:
  • white toolbar (according to theme background color)
  • separators between radio button groups
  • clickable labels instead of buttons

Method browser has many new cool features:
  • Flat and hierarchy view modes. First shows methods as flat list sorted by selectors. And last sorts methods according to class hierarchy 
  • Scopes. You can switch between package, class or system scopes to filter methods. Scopes are inherited from initial tool which opens browser
  • "Child" method browsers are opened in tabs. Click on senders/implementors button do not open new window but instead it shows result in new window tab

To make Calypso default tool evaluate:
ClyBrowser beAllDefault
And little live video:

вторник, 27 декабря 2016 г.

Calypso. New system browser for Pharo

Calypso is a new system browser for Pharo which I developed as required component for general remote IDE project. 
First initial version is done. You can try it by:
Gofer it
smalltalkhubUser: 'Pharo' project: 'Calypso';
#ClySystemBrowser asClass open.
I release Calypso in the state where I finish most design decisions which I think important for project evolution. 
Of course there are a lot of things to do and improve. But browser is fully working. And most Nautilus features are supported. 
Now it is important to community to participate in project and make browser great again :). We need escape from Nautilus pasta code.

Please report any ideas/bugs on github page https://github.com/dionisiydk/Calypso/issues. I already fill it with my wishes.

Next step I will work on documentation/class comments and I will publish couple of posts about Calypso internals

Browser features in many pictures

  • all navigation panes are based on fast table.

  • multiple selection supported everywhere

  • dynamic tabs instead of single source code panel
    • separate tools in tabs to create/edit methods/classes
    • separate tab for class comment
    • extended by plugins

  • dynamic toolbar for hierarchy switches and buttons
    • class refs, senders and implementors again
    • extended by plugins

  • new hierarchies: traits and trait users where available

  • method group view instead of protocols view
    • old "dynamic protocols" as core feature
    • based on method tags as symbols
    • support multiple tags per method (Pharo 7)
    • could provide hierarchy of tags (like in Dolphin)
    • not required star convention for class extension
    • inherited methods group
    • extensions group

  • visibility option for inherited methods

  • methods inherited from traits are not shown by default
    • simple checkbox to show them if exist

  • variable view as special mode for method group view.
    • refactorings directly on shown variables

  • package view is based on class groups
    • "dynamic protocols" for classes
    • based on class tags as symbols
    • support multiple tags per class (Pharo 7)
    • extensions group

  • navigation over hierarchy view automatically selects package of selected class

  • automatic selection of items similar to last selection
    • select method A on class1 then select class2 and method A of class2 will be selected automatically
  • unaccepted edits are not lost when any selection changed
    • tab with modified method stayed "pinned" with special icon
    • same for any other editors (class definition, class comment)
  •  table cells are extendable by plugins
    • ClySUnitBrowserPlugin provides classic icon with test results
    • also icon for package and class tag
  • explicit commands instead of duplicated menu and shortcuts
    • extendable by plugins
There are a lot of things to explain which I will try to do at next steps

пятница, 22 июля 2016 г.

ObjectTravel. A tool to traverse object references

ObjectTravel is a tool to deeply traverse "native" references of objects through instance variables and "array contents".

     Gofer it
          smalltalkhubUser: 'Pharo' project: 'ObjectTravel';

Usage is quite simple: just create traveler instance on your object and send #referencesDo:

      traveler := ObjectTraveler on: (1@2 corner: 3@4).
      traveler referencesDo: [:eachRef | eachRef logCr].

It will print on transcript:


There is convenience method to collect all references:

      traveler collectReferences. "==> an OrderedCollection((1@2) (3@4) 1 2 3 4))

Traveler enumerates each reference in breadth-first direction and visit every reference only once:

      point := 1@2.
      traveler := ObjectTraveler on: {point . #middle. point}.
      traveler collectReferences "==> an OrderedCollection((1@2) #middle (1@2) 1 2)"

"point" is referenced two times. But traveler will go deeply only once. So circles are supported:

      array := Array new: 2.
      array at: 1 put: 1@3.
      array at: 2 put: array.
      traveler := ObjectTraveler on: array.
      traveler collectReferences asArray = { 1@3. array. 1. 3 } "==> true"

You could control traveler to skip particular objects:

      rect := 1@2 corner: 3@4.
      traveler := ObjectTraveler on: rect.
      traveler skip: rect origin.
      traveler collectReferences "==> an OrderedCollection((1@2) (3@4) 3 4)"

(it can be done during enumeration too).

Also you could apply predicate for objects which you want traverse:

      rect :=
      traveler := ObjectTraveler on: {1@2 corner: 3@4. 5@6}. 
      traveler traverseOnly: [:each | each isKindOf: Point]. 
      traveler collectReferences "==> an OrderedCollection((1@2) corner: (3@4) (5@6) 5 6)"

There are another convenient methods:

      traveler countReferences

It will count all references in object graph.

      traveler copyObject

It will make deep copy of original object which is similar to #veryDeepCopy: cyclic references are supported. 

Also traveler could find all paths to given object in object graph:

      root :=  {#one -> #target. {#emptyArray}. {#two -> #target}}.
      traveler := ObjectTraveler on: root.
      traveler findAllPathsTo: #target "==>
           {#one -> #target. root}
            {#two -> #target. {#two -> #target}. root} "

During enumeration traveler could replace visited references:

      rect := 1@2 corner: 3@4.
      traveler := ObjectTraveler on: rect.
      traveler referencesDo: [:each | 
           each isInteger ifTrue: [traveler replaceCurrentReferenceWith: each + 1]].
      rect "==> 2@3 corner: 4@5"

It would be fun project to connect ObjectTraveler to XStreams to make any object "streamable":

      (1@2 corner: 3@4) traveling reading count: [:each | each isInteger].
      (1@2 corner: 3@4) traveling reading get; get; read: 2; rest.
      (1@2 corner: 3@4) traveling writing write: 100@500

четверг, 21 июля 2016 г.

Major Seamless update

I glad to publish new version of Seamless (0.8.2) together with new doc.
It could be loaded by:

       Gofer it
       smalltalkhubUser: 'Pharo' project: 'Seamless';

It works in Pharo 5 and 6. As usually feedback is welcome.

It is complete redesign of original version with the goal to make it more flexible, reliable and simple. (original version was created by Nikolaos Papoulias).

New version introduces new project Basys as foundation for new design.
Basys implements an abstract layer for networks which require bidirectional communication between clients and servers. (details here).
Seamless implements Basys network to organize transparent communication between distributed objects.

To use Seamless SeamlessNetwork should be created on client and server:

       network := SeamlessNetwork new.

To accept connections server should be started:

       network startServerOn: 40422.

Clients could connect to server and retrieve remote environment:

       remotePeer := network remotePeerAt: (TCPAddress ip: #[127 0 0 1] port: 40422).
       remoteSmalltalk := remotePeer remoteEnvironment.

       "or short version"
       remoteSmalltalk := network environmentAt: (TCPAddress localAt: 40422)

remoteSmalltalk here is proxy which delegates any received message to remote object. Remote messages are executed on server which return result back to client. Result can be returned as another proxy or as copy which contains another proxies.

In example result is reference to remote Smalltalk instance. It can access globals from remote environment:

       remoteTranscript := remoteSmalltalk at: #Transcript.
       remoteTranscript open; show: 'remote message'; cr

It will open transcript on server and print text on it.

Arguments of remote message are transferred to server with same logic as message result transferred to client. On server arguments can include proxies and server can send messages to them:

       remoteTranscript print: #(1 2 3 4 5)

Here array will be passed to server as reference. Then on server transcript will interact with it to print it. And as result client will receive messages from server.

Concrete transfer strategy is depends on transferred object. It is specified in method #seamlessDefaultTransferStrategy:

            ^SeamlessTransferStrategy defaultByReference

            ^SeamlessTransferStrategy defaultByValue

Default strategy could be overridden on network level:

       network transferByValue: (Instance of: Point).
       network transferByReference: (Identical to: #specificSymbol)

It allows to tune network for specific application to optimize communication between distributed objects. There are few other transfer strategies which allow minimize remote communication. Most interesting allows properties caching. It will transfer object reference together with specified properties. Following example will configure network to transfer class reference together with name:

       network transferByReference: (Kind to: Class) withCacheFor: #(name)

And then proxy for remote class will return #name from local cache instead of real remote call.

Previously announced project RemoteDebuggingTools uses all these feature to minimize communication between debugger and remote process. It allows to reduce number of remote messages for opening debugger from 800 to 13.

Now another important feature: block evaluation on remote peers:

       remotePeer evaluate: [1 + 2]. "==>3"

Given block is transferred to remote side and evaluated. Result is returned to client. As in other cases it could be proxy or normal object.

Block could use globals. On remote side they will be local globals of this remote environment. Following example will show notification on remote image:

       remotePeer evaluate: [Object inform: 'message from remote image'].

Temps and workspace variables can be used too. They will be transferred according to own strategies:

       | result |
       result := OrderedCollection new.
       remotePeer evaluate: [100 to: 500 do: [:i | result add: i factorial ]].

Non local return is also supported in regular Smalltalk semantics:

       remotePeer evaluate: [1 to: 10 do: [:i | i>5 ifTrue: [^i] ] ]. "==>6"

Also block could be evaluated asynchronously without waiting any result:

       | result |
       result := OrderedCollection new.
       remotePeer evaluateAsync: [result add: 1000 factorial]. "it will not wait result"

Seamless provides integration with GT tools. Remote proxies could be inspected to explore remote objects state with ability to execute remote scripts (doIt, printIt). It is shown on remote debugging demo.

To analyse remote communication Seamless implements special tool SeamlessLogger. It is explained in doc.  Here is little example for following code:

       remoteRect := remotePeer evaluate: [0@0 corner: 2@3].
       remoteRect area. "==>6".
       localRect := 0@0 corner: 5@2.
       remoteRect evaluate: [remoteRect area + localRect area] "==>16"

Statistics will show number of messages, receivers and bytes which was transferred over network in dimension of receiver class or message selector:

At the end I must say about two important issues which will be covered in next versions:

1) No garbage collection

SeamlessNetwork keeps all objects which was transferred by reference. They will never be cleaned while network is live. 
Now It could be done manually by evaluating "network destroy". It will clean all object caches and close all connections. It could be not safe because remote peers could use this objects. Seamless tries to handle it properly with clear errors in such cases. 
In future unused distributed objects will be cleaned automatically.

2) No security: no authorization and encryption. Now for security purpose you need external tools like VPN