python 2.7 - Transform this input file into a list of dictionaries, remove duplicate mongodb -

code:

with open('hocpersons.json.txt') f: arrayoftweets = [] line in f:     arrayoftweets.append(json.loads(line)) 

how remove duplicate data?

here example of data:

arrayoftweets

{u'105824': {u'caucus': u' conservative', u'constituency': u' niagara falls ', u'name': u'hon. rob nicholson', u'province': u' ontario'}, u'105825': {u'caucus': u' conservative', u'constituency': u' niagara falls ', u'name': u'hon. rob nicholson', u'province': u' ontario'}, u'105826': {u'caucus': u' conservative', u'constituency': u' niagara falls ', u'name': u'hon. rob nicholson', u'province': u' ontario'}, u'105835': {u'caucus': u' conservative', u'constituency': u' ottawa west\u2014nepean ', u'name': u'hon. john baird', u'province': u' ontario'}, u'111555': {u'caucus': u'', u'constituency': u'', u'name': u'', u'province': u''}, u'111558': {u'caucus': u' conservative', u'constituency': u' central nova ', u'name': u'hon. peter mackay', u'province': u' nova scotia'}, u'111559': {u'caucus': u'', u'constituency': u'', u'name': u'', u'province': u''}, u'111560': {u'caucus': u' conservative', u'constituency': u' battlefords\u2014lloydminster ', u'name': u'hon. gerry ritz', u'province': u' saskatchewan'}, u'111561': {u'caucus': u' conservative', u'constituency': u' battlefords\u2014lloydminster ', u'name': u'hon. gerry ritz', u'province': u' saskatchewan'}, u'111568': {u'caucus': u' conservative', u'constituency': u' battlefords\u2014lloydminster ', u'name': u'hon. gerry ritz', u'province': u' saskatchewan'}, u'114267': {u'caucus': u' liberal', u'constituency': u' wascana ', u'name': u'hon. ralph goodale', u'province': u' saskatchewan'}, u'128127': {u'caucus': u' ndp', u'constituency': u' london\u2014fanshawe ', u'name': u'irene mathyssen', u'province': u' ontario'}, u'128153': {u'caucus': u'', u'constituency': u'', u'name': u'', u'province': u''}, u'128199': {u'caucus': u'', u'constituency': u'', u'name': u'', u'province': u''}, u'128453': {u'caucus': u'', u'constituency': u'', u'name': u'', u'province': u''}, u'128529': {u'caucus': u'', u'constituency': u'', u'name': u'', u'province': u''}, u'128689': {u'caucus': u' conservative', u'constituency': u' central nova ', u'name': u'hon. peter mackay', u'province': u' nova scotia'}, u'128694': {u'caucus': u' conservative', u'constituency': u' haldimand\u2014norfolk ', u'name': u'hon. diane finley', u'province': u' ontario'}, u'128695': {u'caucus': u'', u'constituency': u'', u'name': u'', u'province': u''}, u'128697': {u'caucus': u'', u'constituency': u'', u'name': u'', u'province': u''}, u'128698': {u'caucus': u' conservative', u'constituency': u' parry sound\u2014muskoka ', u'name': u'hon. tony clement', u'province': u' ontario'}, u'128705': {u'caucus': u' conservative', u'constituency': u' calgary southeast ', u'name': u'hon. jason kenney', u'province': u' alberta'}, u'128706': {u'caucus': u' conservative', u'constituency': u' port moody\u2014westwood\u2014port coquitlam ', u'name': u'hon. james moore', u'province': u' british columbia'}, u'128707': {u'caucus': u' conservative', u'constituency': u' nunavut ', u'name': u'hon. leona aglukkaq', u'province': u' nunavut'}, u'128711': {u'caucus': u' conservative', u'constituency': u' carleton\u2014mississippi mills ', u'name': u'hon. gordon o'connor', u'province': u' ontario'}, u'128713': {u'caucus': u' conservative', u'constituency': u' carleton\u2014mississippi mills ', u'name': u'hon. gordon o'connor', u'province': u' ontario'}, u'128717': {u'caucus': u' conservative', u'constituency': u' blackstrap ', u'name': u'hon. lynne yelich', u'province': u' saskatchewan'}, u'128718': {u'caucus': u' conservative', u'constituency': u' charleswood\u2014st. james\u2014assiniboia ', u'name': u'hon. steven fletcher', u'province': u' manitoba'}, u'128724': {u'caucus': u'',

mongodb python-2.7

share|improve question

edited jun 3 '15 @ 17:12

asked jun 3 '15 @ 0:26 qts 11