2009-02-23 - rspeer
    * Renamed corpus_activity to events_activity.
    * Now that django-voting is included under lib/voting, we need to convert
      Ratings into Votes:

        for r in Rating.objects.all().iterator():
            obj = r.sentence or r.raw_assertion or r.assertion
            score = 0
            if r.score > 0: score=1
            if r.score < 0: score=-1
            Vote.objects.record_vote(obj, r.user, score)
            Event.record_event(obj, r.user, r.activity)

2008-08-28 - rspeer
    * Deleted a Portuguese frame (1170) that had {1} and {2} switched. Its
      predicates and raw predicates (also deleted) were junk with stem1=stem2.

    * Created the English adverb 'never', and used it for negated frames.

    * Stopped using goodness=-1. Negated frames get the 'never' adverb (id=21).

    * Filled in NULL activities on ratings. Most were filled in from sentences,
      but there were a few anomalous ratings by rspeer left:

      update commons_ratings set activity_id = s.activity_id from predicates p,
      sentences s where commons_ratings.activity_id is null and
      commons_ratings.statement_id = p.id and p.sentence_id = s.id;
      UPDATE 12596
      update commons_ratings set activity_id=22 where activity_id is null;
      UPDATE 53

    * I need to learn to stop doing that. Most of these ended up as
      "rubycommons", but some were clear errors. Putting them back the best
      I can:

    ConceptNet=> update commons_ratings set activity_id=null where
    activity_id=5;
    UPDATE 365
    ConceptNet=> update commons_ratings set activity_id=null where
    activity_id=6;
    UPDATE 40
    ConceptNet=> update commons_ratings set activity_id=null where
    activity_id=27;
    UPDATE 3287
    ConceptNet=> update commons_ratings set activity_id=null where
    activity_id=11;
    UPDATE 342


2008-08-27 - rspeer
    * Tracked down raw predicates with their item1 and item2 switched, and
      switched them back


2008-08-18 - kcarnold

	* Set some rubycommons/question sentences:
	rcq=Activity.objects.get(id=31)

	for a in Assertion.objects.filter(source='question', sentence__activity__name='rubycommons').iterator():
	    a.sentence.activity = rcq
	    a.sentence.save()

	* Going to remove the predicates.source column. Mappings are:
SELECT p.source, a.id, a.name FROM predicates p
  INNER JOIN sentences s ON (p.sentence_id=s.id)
  INNER JOIN corpus_activity a ON (s.activity_id=a.id)
  GROUP BY p.source, a.id, a.name
  ORDER BY a.id, source;

        source        | id |                                   name                                   
----------------------+----+--------------------------------------------------------------------------
 csamoa               |  2 | response to diagram
 csamoa               |  5 | template
 csamoa               |  6 | picture description
 csamoa               | 11 | response to picture
 commons manual entry | 16 | commons manual entry
 csamoa               | 16 | commons manual entry
 csamoa               | 20 | pycommons/question
 pycommons/question   | 20 | pycommons/question
 commons              | 22 | rubycommons
 csamoa               | 22 | rubycommons (assertions 1494905,1494906,1494904,1494892,1494893,1494896,1494907,1494908,1494909)
 csamoa               | 24 | rubycommons manual entry
 csamoa               | 27 | omcs1, possibly free text
 csamoa               | 28 | OMCS no Brasil
 is-a                 | 29 | is-a cleanup
 csamoa               | 31 | rubycommons/question (some questions are confused as just "rubycommons")*
 question             | 31 | rubycommons/question (some questions are confused as just "rubycommons")

* just assertion 1494905


2008-08-15 - kcarnold

	* Set "predicate" foreign key on raw assertions generated by csamoa
	that were the last to update the assertion (the easy case).
	
	ConceptNet=> UPDATE parsing_rawpredicate r SET predicate_id=p.id
	ConceptNet-> FROM predicates p WHERE p.raw_id=r.id AND r.predicate_id is null;
	UPDATE 88214

	* Fix where probably my script incorrectly set the predicate. I think
	these are all cases where the same sentence got parsed into different
	assertions between different batches.
	ConceptNet=> UPDATE parsing_rawpredicate r SET predicate_id=p.id
	ConceptNet-> FROM predicates p WHERE p.raw_id=r.id AND r.predicate_id != p.id;
	UPDATE 4055	
	
2008-08-07 - rspeer
        * Found csamoa other-ratings:
        
          a, c = Activity.objects.get_or_create(name="csamoa other-rating")
          for b in Batch.objects.filter(updated__lt=datetime(2007,10,1)):
            for rating in Rating.objects.filter(updated_on__gte=b.created, updated_on__lte=b.updated):
                if rating.updated_on.microsecond != 0:
                    rating.activity = a
                        rating.save()

        * And that got everything but the manual ratings!
        

2008-08-06 - rspeer
        * Reconciled activities with sources. Sources are now obsolete.
        * Junk activities were all moved into 'junk'.
        * New activities: 'rubycommons', 'rubycommons manual entry',
          'Link Grammar is-a', 'junk', 'is-a cleanup', 'unknown'
          * 'rubycommons' if source='question', activity='default'
          * 'rubycommons manual entry' if source='csamoa', activity='default'
            but it didn't come from a csamoa run
          * 'Link Grammar is-a' if source='is-a'
          * 'junk' if it was one of the accidental one-off activities created
            by omcs1 users typing two asterisks
          * there is nothing in 'unknown'
        * Some duplicate assertions had been created by the link grammar
          process. These were cleaned up on August 14, 2007. The assertions
          and ratings that were produced on August 14 are marked with
          "is-a cleanup" as the activity.
          
          a, c = Activity.objects.get_or_create(name='is-a cleanup')
          Rating.objects.filter(updated_on__gt='2007-08-14', updated_on__lt='2007-08-15').update(activity=a)
          Sentence.objects.filter(created_on__gt='2007-08-14', created_on__lt='2007-08-15').update(activity=a)

2008-02-28 - kcarnold
	* set activity_id on Ratings with the following SQL:
	UPDATE commons_ratings SET activity_id=17 WHERE rating_value_id >= 13;
	* set activity on Ratings for new manual entries with the following Python:
	a=Activity.objects.get(name='commons manual entry')
	for assertion in Assertion.objects.filter(sentence__activity=a):
	    rating = assertion.get_rating(assertion.creator)
	    if rating is not None and rating.rating_value_id == 13: # Good
	        rating.activity = a
	        rating.save()

	* set activity on Ratings for user test users:
	for user in User.objects.filter(username__startswith='temp'):
	    if user.password != user.username[4:]: continue
	    for rating in user.rating_set.all():
	        rating.activity = a
	        rating.save()

	* set activity on Ratings for csamoa automatic self-ratings:
	a, c = Activity.objects.get_or_create(name='csamoa self-rating (probably)')
	for b in Batch.objects.filter(updated__lt=datetime(2007,10,1)):
	    for rating in Rating.objects.filter(updated_on__gte=b.created, updated_on__lte=b.updated):
	        if rating.activity_id: continue
	        if rating.statement.creator != rating.user: continue
	        rating.activity = a
	        rating.save()
	(should have checked that statement source was csamoa and the rating was actually Generally True)

	* set activity on Ratings for commons automatic self-ratings:
	a, c = Activity.objects.get_or_create(name='commons self-rating')
	for assertion in Assertion.objects.filter(source='commons'):
	    rating = assertion.get_rating(assertion.creator)
	(HOLDING OFF on this until I figure out how 'source' got set in Ruby Commons)
	

2008-02-07 - kcarnold
	* added activity_id to Rating.
	
2008-02-05 - kcarnold
	* dropped column depth from Predicate. (was all 0)
	* dropped column old_id from Sentence. (was all 0)
	
2008-02-04 - kcarnold
	* added Frequency model, and frequency_id to frame.
